[egenix-users] Using mxodbc with SQL Server 2012

M.-A. Lemburg mal at egenix.com
Wed Sep 2 23:55:47 CEST 2015 

On 02.09.2015 22:14, John Anderson wrote:
> On Wed, Sep 2, 2015 at 1:05 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> On 02.09.2015 20:36, John Anderson wrote:
>>> Hey, We currently use pymssql and it was recommended that we checkout
>>> mxODBC again (its been a few years) and even though most of our projects
>>> require python 3 we have a few legacy python applications that are still
>> on
>>> python 2.7 so I thought I would give it a try.
>>>
>>> I'm currently getting this error:
>>>
>>> mx.ODBC.Error.OperationalError: ('08003', 0, '[unixODBC][Driver
>>> Manager]Connnection does not exist', 13710)
>>
>> This is a typical error message you get when the connection is found
>> by the ODBC manager, i.e. it is configured in the odbc.ini file,
>> but it cannot load it.
>>
>> This can be a permission problem or a linker problem. You can
>> try this to check:
>>
>> ldd /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
>>
>> Also see our documentation for FreeTDS:
>>
>> http://www.egenix.com/products/python/mxODBC/doc/#_Toc427692966
>>
>> A note on FreeTDS:
>>
>> You should really use the MS SQL Server Native Client for
>> Linux, if you can. It is much more feature complete compared
>> to FreeTDS and also much easier to configure:
>>
>> https://msdn.microsoft.com/en-us/library/hh568451%28v=sql.110%29.aspx
>>
>> These are the instructions from our documentation:
>>
>> http://www.egenix.com/products/python/mxODBC/doc/#_Toc427692959
>>
>>
> I figured it is some type of config problem but its really odd because
> SQLAlchemy can make it work just fine, this lets me connect and query:
> 
> from sqlalchemy import create_engine
> engine = create_engine('mssql+mxodbc://<secret>:<secret>@mt1')

SA translates the connect to a call like this:

db = mx.Connect('mt1', '<UID>', '<PWD>')

It's possible that naming both the DSN and the TDS server using
the same identifer confuses some part in the setup when using
DriverConnect(). The latter is more flexible, though, since it
allows adding more connection options.

An ODBC trace will usually show more information about what exactly
is not working. The mxODBC documentation explains how this
can be enabled (but I see below that you have that enabled already).

> I'm looking into the MS SQL Server Native client again as well, we tried it
> when they first announced it and it would segfault with any significant
> load and they mostly support Red Hat instead of Ubuntu.  I want to compare
> all variations of it.
> 
> The reason we are finally looking at other drivers is because the pymssql
> driver isn't quite fast enough, when selecting 2 million records it takes
> about 10 seconds:
> 
> $ time python pymssql_query.py
> 2148068
> 
> real 0m11.138s
> user 0m9.606s
> sys 0m0.724s
> 
> With mxODBC + SQLAlchemy (since I can't get mxodbc to work without SA):
> 
> $ time python mxodbc_sa.py
> 2148068
> 
> real 0m49.407s
> user 0m19.013s
> sys 0m27.161s
> 
> So it looks like mxodbc is significantly slower than pymssql even though
> both are utilizing FreeTDS behind the scenes.

The SA code is likely doing a lot more on top of just running
the query. You also have tracing enabled, which slows down things
a lot.

There are also several options to optimize queries to SQL Server
using mxODBC (e.g. using read-only connections, different
transaction isolations, direct execution, NO COUNT).

We also have the mxODBC Connect product which removes the need for
a Unix ODBC driver altogether and uses the MS driver on the
Windows database server to directly communicate via shared
memory with the database kernel. mxODBC Connect takes care of
the client-server communication and is optimized for the
Python DB-API way of fetching data.

> I'm going to try the native driver next and then I'll let you know.
> 
> Here is the output of the ldd command:
> 
> ldd /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
> linux-vdso.so.1 =>  (0x00007ffec93f6000)
> libodbcinst.so.2 => /usr/lib/x86_64-linux-gnu/libodbcinst.so.2
> (0x00007ff192bcd000)
> libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
> (0x00007ff192985000)
> libgnutls-deb0.so.28 => /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28
> (0x00007ff19266a000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007ff19244c000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff192082000)
> libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007ff191e77000)
> libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007ff191ba8000)
> libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
> (0x00007ff191977000)
> libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2
> (0x00007ff191772000)
> libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
> (0x00007ff191567000)
> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ff19134c000)
> libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
> (0x00007ff191105000)
> libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6
> (0x00007ff190ef1000)
> libnettle.so.4 => /usr/lib/x86_64-linux-gnu/libnettle.so.4
> (0x00007ff190cc0000)
> libhogweed.so.2 => /usr/lib/x86_64-linux-gnu/libhogweed.so.2
> (0x00007ff190a91000)
> libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007ff190811000)
> /lib64/ld-linux-x86-64.so.2 (0x00007ff193066000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff19060d000)
> libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
> (0x00007ff190408000)
> libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007ff1901ec000)
> libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007ff18ffe3000)

Looks fine, so it's probably not related to the linker.

>>> with this code:
>>>
>>> import mx.ODBC.unixODBC as mx
>>> db = mx.DriverConnect('DSN=mt1;UID=<secret>;PWD=<secret>')
>>> c = db.cursor()
>>> c.tables()
>>> res = c.fetchall()
>>>
>>> and I'm wondering might cause that because I can use isql/tsql and it
>> works:
>>>
>>> tsql -S mt1 -U <secret> -P <secret>
>>> isql mt1 <secret> <secret>
>>>
>>> osql is the only one that doesn't work:
>>>
>>>
>>> osql -S mt1 -U <secret> -P <secret>
>>>
>>> checking shared odbc libraries linked to isql for default directories...
>>> strings: '': No such file
>>> trying /tmp/sql ... no
>>> trying /tmp/sql ... no
>>> trying /etc ... OK
>>> checking odbc.ini files
>>> reading /home/sontek/.odbc.ini
>>> [mt1] not found in /home/sontek/.odbc.ini
>>> reading /etc/odbc.ini
>>> [mt1] found in /etc/odbc.ini
>>> found this section:
>>> [mt1]
>>> Driver          = FreeTDS
>>> Description     = ODBC connection via FreeTDS
>>> Trace           = Yes
>>> TraceFile       = /tmp/sql.log
>>> ServerName      = mt1
>>> Database        = SM_AccountsNew
>>> looking for driver for DSN [mt1] in /etc/odbc.ini
>>>   found driver line: " Driver          = FreeTDS"
>>>   driver "FreeTDS" found for [mt1] in odbc.ini
>>> found driver named "FreeTDS"
>>> "FreeTDS" is not an executable file
>>> looking for entry named [FreeTDS] in /etc/odbcinst.ini
>>>   found driver line: " Driver          =
>>> /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so"
>>>   found driver /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so for [FreeTDS]
>>> in odbcinst.ini
>>> /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so is not an executable file
>>> osql: error: no driver found for mt1
>>>
>>>
>>> Anyone know what I might be missing?
>>>
>>> Here is my FreeTDS:
>>>
>>> [mt1]
>>>         host = mt1-db01.corp.surveymonkey.com
>>>         port = 1433
>>>         tds version = 7.3
>>>         client charset = UTF-8
>>>
>>> odbc.ini:
>>>
>>> [mt1]
>>> Driver          = FreeTDS
>>> Description     = ODBC connection via FreeTDS
>>> Trace           = Yes
>>> TraceFile       = /tmp/sql.log
>>> ServerName      = mt1
>>> Database        = SM_AccountsNew
>>>
>>>
>>> odbcinst.ini:
>>> [FreeTDS]
>>> Description     = TDS
>>> Driver          = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
>>> Startup         = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
>>> UsageCount      = 1
>>> Threading       = 2
>>>
>>>
>>>
>>>
>>> _______________________________________________________________________
>>> eGenix.com User Mailing List                     http://www.egenix.com/
>>> https://www.egenix.com/mailman/listinfo/egenix-users
>>>
>>
>> --
>> Marc-Andre Lemburg
>> eGenix.com
>>
>> Professional Python Services directly from the Source  (#1, Sep 02 2015)
>>>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
>> ________________________________________________________________________
>> 2015-08-27: Released eGenix mx Base 3.2.9 ...     http://egenix.com/go83
>>
>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>>
>>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>            Registered at Amtsgericht Duesseldorf: HRB 46611
>>                http://www.egenix.com/company/contact/
>>
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 02 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2015-08-27: Released eGenix mx Base 3.2.9 ...     http://egenix.com/go83

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

More information about the egenix-users mailing list