Is it possible to add/install/import third party Python modules - to extend the functionality of the User Defined Transform (UDT)?
Currently UDT only supports the out-of-the-box Python 2.6 functionality and I find myself re-inventing the wheel over again, despite third-party modules being available that would take care of this.
I suspect that all the code in the UDT gets parsed by the Python compiler externally to Data Services - so once the modules have been installed, it should be possible to import them as part of the UDT?
I actually use a lib folder on our SAN that (most) all my python resides.
That path is added to the python search path so when you do an import it also searches that folder.
I dont have direct access to the python install folder on the job server, so this helps me out a lot. I still need our admin to install new modules, but I can put my code where I can easily access it.
Thanks. I managed to get this working my ensuring that the Python directory and underlying Script/Lib directories are all added to the path on the server. That way, I can use any external library by just referencing it in my Python code within the UDT. (e.g. import oauth2 etc.)
By adding the paths, I can now also use PIP to add new modules etc.
As far as I know between the 2.x and 3.x branch there is not much backwards compatability. So 3.x is not definitely better based on the version number.
But jlynn73 can explain this better, I think he is the expert on Python here. I only copy pasted some things together until it worked… Sometimes you have requirements that are not possible in BODS, or not in a performing way…
Interesting enough I am considering moving the python code out of BODS. Because it comes with the DQ license and that is a heavy license. So writing a flat file to the disk, call python via the command line will also work. Does it give overhead? Probably, but then you could jump to any version of Python you want and implement more fancy stuff as parallel processing.
Setting the DOP of the DF with a DQ Python component also works, but I have the feeling it is less efficient.
There is no fear of migrating that which is not inside DataServices.
I spent lot of time converting python scripts into DS scripting language, or out converted it. I believe I shaved 60% off our merge purge match processing times by taking the python out. Using python isnt such a big deal until you start pushing millions of records through it, combined with several other queries… and it becomes pretty inefficient.
If you are planning on using it, and are using a large dictionary/complex data structures… a neat trick is to pickle the data structure to a file. Then you can work against the data structure outside of DS, finish the code and port it back in to DS.
I noticed that pip was not installed on the job server but luckily found two alternatives:
if you have admin right to the server - extract the 3rd party module to a folder and then run python setup.py install for each module you need installed
without admin rights but with write access to the Python’s install folder on the server, first install the 3rd party module on your local machine and then copy and paste the relevant package from your Python27\Lib\site-packages to the server’s Python’s \Lib\site-packages folder
Re: Python 3, I read it will form part of DS 4.2 SP13 when that becomes available.
(BOB member since 2007-01-10)
(BOB member since 2009-10-27)
(BOB member since 2007-09-12)
(BOB member since 2012-03-20)