Simple python config

Often when I'm writing python code I'll be setting some values and think to myself "Gee, these values should really be stored in a config file". That thought is often followed by "Nah, too much effort. This is just a quick script anyhow". If there was a way of making using config files almost zero effort then I'd be more likely to use them and not have to deal with my "quick script" now being a couple of hundred lines of code and having no easy way of changing the configuration.

Python's ConfigParser module can do most of the work, but there are still some things that make it just that little bit of effort to use.

• It doesn't know/care about types. Everything is just a string. You have to consciously call getint or getfloat etc depending on what type you want.
• I'd prefer to just have a simple dictionary with all my values in it - config['section:key'] looks neater than config.get('section', 'key') (to me anyhow)
• It can't deal with config files that don't have a section in them
• If I'm working on amazingProgram.py I'd like it to default to using amazingProgram.cfg without having to explicitly tell it.

I've written a fairly simple function to get around these issues. It uses ConfigParser to read the config file, and turns it into a simple dictionary. For each value it determines if it is a int, float, bool (denoted by any of 'true', 'yes', 'on', 'false', 'no', 'off') or if none of these leaves it as a string.

You can tell it which config file to read, but if none is given it will take the name of the file containing the __main__ function, and replace the '.py' with '.cfg'. Using it is as simple as

import simpleConfig

config = simpleConfig.parseConfig()

So let's say I have a config file called test.cfg which contains

host = localhost
port = 8080
delay = 1.0
use_tcp = true

And I make a test.py with

import simpleConfig

config = simpleConfig.parseConfig()

if config['use_tcp']:
        print config['host'], type(config['host'])
        print config['port'], type(config['port'])
        print config['delay'], type(config['delay'])

When I run it I get

[rlawther@wagner tmp]$ python test.py
localhost <type 'str'>
8080 <type 'int'>
1.0 <type 'float'>

So hopefully, next time I think I should put something in a config file I'll use this rather than not bothering.

Building pyliblo on Windows using Visual C++ 2010

Python's great, I quite like liblo, and sometimes I'm forced to use Windows. I couldn't use mingw, so these 3 facts lead me to trying to get pyliblo to build on Windows using Visual C++ 2010. Feel free to follow along at home if you will ...

This will show all the errors you encounter along the way and how to overcome them. So hopefully parts of this may apply to other python modules out there as well.

Get pthreads for windows

liblo uses pthreads, so you'll need to get that first. I got mine from here, but I guess just get whatever happens to be the latest when you read this. Get the .zip version as it has precompiled DLLs in there, unless you particularly feel like build yourself. Extract the zip file to a scratch directory somewhere - I'm using d:\tmp

Build liblo

We need to build the liblo DLLs for pyliblo to use. So ...

• If you haven't already, you'll need to install Visual C++ 2010. I installed the express edition (for free) from here
• OK, now get the liblo source and extract it also to d:\tmp
• Your scratch dir should now look like this



Folders after extacting pthreads and liblo

• Open up a command shell (Start -> Run -> cmd) and change to the liblo-0.26\build directory. Run the command premake4.exe vs2008 :

  D:\tmp\liblo-0.26\build>premake4.exe vs2008
  Building configurations...
  Running action 'vs2008'...
  Generating vs2008/liblo.sln...
  Generating vs2008/liblo.vcproj...
  Generating vs2008/testlo.vcproj...
  Generating vs2008/subtest.vcproj...
  Done.
  

• This has now generated our solution file for us. Back in Windows Explorer, double click on the D:\tmp\liblo-0.26\build\vs2008\liblo.sln file.
• Because this file was generated for Visual Studio 2008, it will want to convert it to the 2010 format. Just click Next, Next, Finish.
•  Now inside Visual Studio, first we want to change the Active Solution Configuration from DebugDLL to ReleaseDLL. There's a drop-down box near the top of the screen to do this



Selecting the ReleaseDLL solution configuration




• OK, so let's try to build these liblo DLLs. Right click on the "liblo" text in the "Solution Explorer" and select "build".
• Look at the output of the build command in the output window at the bottom. You should see that it failed with errors looking like

  d:\tmp\liblo-0.26\src\lo_types_internal.h(31): fatal error C1083:
  Cannot open include file: 'pthread.h': No such file or directory

• OK, so it can't find the pthread headers, we'll need to let it know where they are. So, right click on the liblo solution again, and select the properties option.  In the properties window go to Configuration Properties -> C/C++ -> General  and find the Additional Include Directories entry. We need to add ;..\..\..\Pre-built.2\include. Note the semicolon at the start to separate it from the previous directory



Adding an include directory

• Try and build again. This time it will fail with

  LINK : fatal error LNK1104:
  cannot open file 'pthreadVC2.lib'

  OK, so it compiled fine, it's the linker that now complaining. We need to tell the linker where it can find the lib file. Again right click on the liblo solution and select properties. This time we want to edit the Configuration Properties -> Linker -> General -> Additional Library Directories entry and add the string ;..\..\..\Pre-built.2\lib\x86 (building for 64 bit OSs is left as an excercie for the reader)
• Build again - this time it should work and you'll see that ever so pleasing output

======= Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped =======

• Great, now you've got liblo DLLs - they'll be in the D:\tmp\liblo-0.26\lib\ReleaseDLL folder.
 

Build pyliblo

OK so this is the part we've all been waiting for, we actually get to build pyliblo.

• Grab pyliblo from here and extract as before.
• Again we'll get another command shell and try to compile as we usually would for a python module by running python setup.py build

D:\tmp\pyliblo-0.9.1>python setup.py build
running build
running build_ext
building 'liblo' extension
error: Unable to find vcvarsall.bat

• Hmm, okay, python is set up to use Visual Studio 9 (ie. 2008) so it can't find the 2010 files. We can fix that by running

  SET VS90COMNTOOLS=%VS100COMNTOOLS%

• Try again:

  D:\tmp\pyliblo-0.9.1>python setup.py build
  running build
  running build_ext
  building 'liblo' extension
  creating build
  creating build\temp.win32-2.7
  creating build\temp.win32-2.7\Release
  creating build\temp.win32-2.7\Release\src
  D:\apps\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /
  DNDEBUG -Id:\python\python27\include -Id:\python\python27\PC /Tcsrc/liblo.c /Fob
  uild\temp.win32-2.7\Release\src/liblo.obj -fno-strict-aliasing -Werror-implicit-
  function-declaration -Wfatal-errors
  cl : Command line error D8021 : invalid numeric argument '/Werror-implicit-funct
  ion-declaration'
  error: command '"D:\apps\Microsoft Visual Studio 10.0\VC\BIN\cl.exe"' failed wit
  h exit status 2
  

• At least it's found the compiler, but now we're getting compiler issues. There's a bunch of flags being passed to the compiler that it doesn't understand (such as -Werror-implicit-function-declaration) ... we'll just remove them. Open up setup.py in your favourite text editor. Look for the flags that cause the issues ... there they are

  ext_modules = [
      Extension(
          'liblo',
          [use_cython and 'src/liblo.pyx' or 'src/liblo.c'],
          extra_compile_args = [
              '-fno-strict-aliasing',
              '-Werror-implicit-function-declaration',
              '-Wfatal-errors',
          ],
          libraries = ['lo']
      )
  ]
  

  Just remove all the extra_compile_args ...

  ext_modules = [
      Extension(
          'liblo',
          [use_cython and 'src/liblo.pyx' or 'src/liblo.c'],
          extra_compile_args = [],
          libraries = ['lo']
      )
  ]

• Sweet, let's try to compile that.

  D:\tmp\pyliblo-0.9.1>python setup.py build
  running build
  running build_ext
  building 'liblo' extension
  D:\apps\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /
  DNDEBUG -Id:\python\python27\include -Id:\python\python27\PC /Tcsrc/liblo.c /Fob
  uild\temp.win32-2.7\Release\src/liblo.obj
  liblo.c
  src/liblo.c(221) : fatal error C1083: Cannot open include file: 'lo/lo.h': No su
  ch file or directory
  error: command '"D:\apps\Microsoft Visual Studio 10.0\VC\BIN\cl.exe"' failed wit
  h exit status 2

• Right, it can't find the liblo headers. Let's show it where to look. We'll point it at the pthread headers too while we're at it.

  ext_modules = [
      Extension(
          'liblo',
          [use_cython and 'src/liblo.pyx' or 'src/liblo.c'],
          extra_compile_args = ['-ID:\\tmp\\liblo-0.26', '-ID:\\tmp\\Pre-built.2\\include'],
          libraries = ['lo']
      )
  ]
  

• Let's see how this goes

  D:\tmp\pyliblo-0.9.1>python setup.py build
  running build
  running build_ext
  building 'liblo' extension
  D:\apps\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /
  DNDEBUG -Id:\python\python27\include -Id:\python\python27\PC /Tcsrc/liblo.c /Fob
  uild\temp.win32-2.7\Release\src/liblo.obj -ID:\tmp\liblo-0.26
  liblo.c
  D:\tmp\liblo-0.26\lo/lo_endian.h(34) : fatal error C1083: Cannot open include fi
  le: 'netinet/in.h': No such file or directory
  error: command '"D:\apps\Microsoft Visual Studio 10.0\VC\BIN\cl.exe"' failed wit
  h exit status 2
  

• Now that's interesting. It's looking for netinet/in.h which is a POSIX (ie. unix / linux type) thing. Not surprising it didn't find it on a Windows system. Why is it looking for it? Let's open up D:\tmp\liblo-0.26\lo/lo_endian.h(34) from the error message and see what's going on

  #ifdef WIN32
  #include <winsock2.h>
  #include <ws2tcpip.h>
  #else
  #include <netinet/in.h>
  #endif
  

  So it's basically saying if it's a Windows machine use winsock2, otherwise use netinet. This makes sense, but doesn't appear to be working. For some reason WIN32 isn't being defined - let's define it ourselves with some more extra_compile_args:

          extra_compile_args = ['-ID:\\tmp\\liblo-0.26', '-ID:\\tmp\\Pre-built.2\\include', '-DWIN32'],
  
  

• Let's try now

  D:\tmp\pyliblo-0.9.1>python setup.py build
  running build
  running build_ext
  building 'liblo' extension
  D:\apps\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /
  DNDEBUG -Id:\python\python27\include -Id:\python\python27\PC /Tcsrc/liblo.c /Fob
  uild\temp.win32-2.7\Release\src/liblo.obj -ID:\tmp\liblo-0.26 -DWIN32 -ID:\tmp\P
  re-built.2\include
  liblo.c
  src/liblo.c(7477) : warning C4244: 'function' : conversion from 'double' to 'flo
  at', possible loss of data
  src/liblo.c(8002) : warning C4244: 'function' : conversion from 'double' to 'flo
  at', possible loss of data
  creating build\lib.win32-2.7
  D:\apps\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:N
  O /LIBPATH:d:\python\python27\libs /LIBPATH:d:\python\python27\PCbuild lo.lib /E
  XPORT:initliblo build\temp.win32-2.7\Release\src/liblo.obj /OUT:build\lib.win32-
  2.7\liblo.pyd /IMPLIB:build\temp.win32-2.7\Release\src\liblo.lib /MANIFESTFILE:b
  uild\temp.win32-2.7\Release\src\liblo.pyd.manifest
  LINK : fatal error LNK1181: cannot open input file 'lo.lib'
  error: command '"D:\apps\Microsoft Visual Studio 10.0\VC\BIN\link.exe"' failed w
  ith exit status 1181
  

• So it's compiled now, but fails to link. It wants to link against lo.lib but it can't find it. There's two reasons for that - firstly it doesn't know where to look. We'll add some extra_link_args to show it where our .lib file is. Secondly, it's got the wrong filename - *nix systems will append "lib" to the front of library names, so we'll have to do that manually here and change the libraries option from 'lo' to 'liblo'

  ext_modules = [
      Extension(
          'liblo',
          [use_cython and 'src/liblo.pyx' or 'src/liblo.c'],
          extra_compile_args = ['-ID:\\tmp\\liblo-0.26', '-DWIN32', '-ID:\\tmp\\Pre-built.2\\include'],
   extra_link_args = ['/LIBPATH:D:\\tmp\\liblo-0.26\\lib\\ReleaseDLL',
          libraries = ['liblo']
      )
  ]
  

• Surely this time ...

  D:\tmp\pyliblo-0.9.1>python setup.py build
  running build
  running build_ext
  building 'liblo' extension
  D:\apps\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /
  DNDEBUG -Id:\python\python27\include -Id:\python\python27\PC /Tcsrc/liblo.c /Fob
  uild\temp.win32-2.7\Release\src/liblo.obj -ID:\tmp\liblo-0.26 -DWIN32 -ID:\tmp\P
  re-built.2\include
  liblo.c
  src/liblo.c(7477) : warning C4244: 'function' : conversion from 'double' to 'flo
  at', possible loss of data
  src/liblo.c(8002) : warning C4244: 'function' : conversion from 'double' to 'flo
  at', possible loss of data
  D:\apps\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:N
  O /LIBPATH:d:\python\python27\libs /LIBPATH:d:\python\python27\PCbuild liblo.lib
   /EXPORT:initliblo build\temp.win32-2.7\Release\src/liblo.obj /OUT:build\lib.win
  32-2.7\liblo.pyd /IMPLIB:build\temp.win32-2.7\Release\src\liblo.lib /MANIFESTFIL
  E:build\temp.win32-2.7\Release\src\liblo.pyd.manifest /LIBPATH:D:\tmp\liblo-0.26
  \lib\ReleaseDLL
     Creating library build\temp.win32-2.7\Release\src\liblo.lib and object build\
  temp.win32-2.7\Release\src\liblo.exp
  C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\mt.exe -nologo -manifest build
  \temp.win32-2.7\Release\src\liblo.pyd.manifest -outputresource:build\lib.win32-2
  .7\liblo.pyd;2
  
  build\temp.win32-2.7\Release\src\liblo.pyd.manifest : general error c1010070: Fa
  iled to load and parse the manifest. The system cannot find the file specified.
  error: command 'mt.exe' failed with exit status 31
  

• Now it's complaining that it can't find the manifest file that should have been created in the linking stage. We'll need to explicitly tell the linker to generate the manifest:

  ext_modules = [
      Extension(
          'liblo',
          [use_cython and 'src/liblo.pyx' or 'src/liblo.c'],
          extra_compile_args = ['-ID:\\tmp\\liblo-0.26', '-DWIN32', '-ID:\\tmp\\Pre-built.2\\include'],
   extra_link_args = ['/LIBPATH:D:\\tmp\\liblo-0.26\\lib\\ReleaseDLL', '/MANIFEST'],
          libraries = ['liblo']
      )
  ]
  

• And ...

  D:\tmp\pyliblo-0.9.1>python setup.py build
  running build
  running build_ext
  building 'liblo' extension
  D:\apps\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /
  DNDEBUG -Id:\python\python27\include -Id:\python\python27\PC /Tcsrc/liblo.c /Fob
  uild\temp.win32-2.7\Release\src/liblo.obj -ID:\tmp\liblo-0.26 -DWIN32 -ID:\tmp\P
  re-built.2\include
  liblo.c
  src/liblo.c(7477) : warning C4244: 'function' : conversion from 'double' to 'flo
  at', possible loss of data
  src/liblo.c(8002) : warning C4244: 'function' : conversion from 'double' to 'flo
  at', possible loss of data
  creating build\lib.win32-2.7
  D:\apps\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:N
  O /LIBPATH:d:\python\python27\libs /LIBPATH:d:\python\python27\PCbuild liblo.lib
   /EXPORT:initliblo build\temp.win32-2.7\Release\src/liblo.obj /OUT:build\lib.win
  32-2.7\liblo.pyd /IMPLIB:build\temp.win32-2.7\Release\src\liblo.lib /MANIFESTFIL
  E:build\temp.win32-2.7\Release\src\liblo.pyd.manifest /LIBPATH:D:\tmp\liblo-0.26
  \lib\ReleaseDLL /MANIFEST
     Creating library build\temp.win32-2.7\Release\src\liblo.lib and object build\
  temp.win32-2.7\Release\src\liblo.exp
  C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\mt.exe -nologo -manifest build
  \temp.win32-2.7\Release\src\liblo.pyd.manifest -outputresource:build\lib.win32-2
  .7\liblo.pyd;2
  running build_scripts
  creating build\scripts-2.7
  copying and adjusting scripts\send_osc.py -> build\scripts-2.7
  copying and adjusting scripts\dump_osc.py -> build\scripts-2.7
  renaming build\scripts-2.7\send_osc.py -> build\scripts-2.7\send_osc
  renaming build\scripts-2.7\dump_osc.py -> build\scripts-2.7\dump_osc
  

• Yay, it built! (About bloody time ...)

Use it!

Now we're ready to check that pyliblo works and get to using it. Make a new scratch directory and copy D:\tmp\pyliblo-0.9.1\build\lib.win32-2.7\liblo.pyd, D:\tmp\liblo-0.26\lib\ReleaseDLL\liblo.dll and D:\tmp\Pre-built.2\dll\x86\pthreadVC2.dll into it. Run python from that directory and try to import liblo:

D:\tmp\test>python
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> import liblo
>>> dir(liblo)
['Address', 'AddressError', 'Bundle', 'Message', 'Server', 'ServerError', 'Serve
rThread', 'TCP', 'UDP', 'UNIX', '_Blob', '_ServerBase', '__builtins__', '__doc__
', '__file__', '__name__', '__package__', '__test__', '__version__', '_inspect',
 '_weakref', '_weakref_method', 'make_method', 'send', 'struct', 'time']
>>>

We're away!