Re: PyGObject (3.4.0) - importing overrides modules from different directories

[Simon Feltman <s feltman gmail com>]

Hi Lukas,

Yes, and my apologies, I did not read your original email close enough
to realize this is what you are running into. Before going the route
of symlinks it may be worth while to see if .pkg files or
pkg_resources.declare_namespace could help the situation (I haven't
had the time to look into either). We could also start to turn
gi/overrides/ into a pure namespace package by removing code from the
__init__.py (I think we should do this in any case) but this will take
a deprecation cycle. The following PEP is also informative if you
haven't gone through it already:
http://www.python.org/dev/peps/pep-0420/
At this point, creating a bug for tracking this so we can start
trading and reviewing patches more easily would be nice.

-Simon

On Wed, Oct 31, 2012 at 3:35 AM, Lukas Vacek <lucas vacek gmail com> wrote:
> Simon,
>
> I see you had ran into the problem with extend_path and code in
> __init__.py already (comment 26 in the bug report you posted earlier).
> Do you think it is a good practice for pygobject overrides module
> developers to create a symlink from __init__.py in their installed
> module to __init__.py in pygobject overrides/ directory?
>
> Lukas
>
> On Fri, Oct 26, 2012 at 1:25 PM, Lukas Vacek <lucas vacek gmail com> wrote:
>> Hey Simon,
>>
>> Thanks for pointing me to the bug report. Yes, it is possible to load
>> overrides from different paths already. This goes a bit further and
>> allows loading overrides from different paths no matter the order of
>> sys.path.
>>
>> Let's have a look at gst-python as it is the subject of the bug
>> report: if you run PYTHONPATH=path-to-gst-python python -c "import gi;
>> print(str(dir(gi)))" you can see gi module wasn't loaded properly.
>> That's because __init__.py in pygobject overrides contains other code
>> than just calling extend_path from pkgutil. Indeed you can use
>> PYTHONPATH=path-to-pygobject:path-to-gst-python and all is good.
>>
>> This might seem like a trivial issue. Well, it's nothing major, but it
>> causes confusion - for example, the two issues in pitivi I linked in
>> my first email stems from this.
>>
>> Cheers,
>> Lukas
>>
>> On Fri, Oct 26, 2012 at 12:53 PM, Simon Feltman <s feltman gmail com> wrote:
>>> Hi Lukas,
>>>
>>> You should already be able to use overrides in different paths
>>> searchable by python (anything that shows up in sys.path). The trick
>>> is you need to use pkgutil within each of the __init__.py files within
>>> your custom gi/overrides folders. There are more details which may be
>>> a helpful in this bug report:
>>> https://bugzilla.gnome.org/show_bug.cgi?id=680913
>>>
>>> -Simon
>>>
>>> On Fri, Oct 26, 2012 at 4:23 AM, Lukas Vacek <lucas vacek gmail com> wrote:
>>>> I see the last mail didn't catch much attention. Anyway, I have come
>>>> up with two possible solutions to the problem.
>>>>
>>>> 1) Use imp instead of __import__
>>>> Just look for a file "gi/overrides/Module_name.py" in sys.path and
>>>> don't worry about __init__.py (overrides module can't use __init__.py
>>>> anyway as they clash with pygobject's overrides' init which is the
>>>> root cause of the whole problem), then load it with imp. It feels a
>>>> bit untidy but works fine. This is the attached
>>>> pygobject-3.4.0-overrides-load-py-files-instead-of-modules.patch.
>>>> Indeed this should look for other supported suffixes than .py as well.
>>>>
>>>> 2) Ask overrides module developers to symlink __init__.py
>>>> This solution feels cleaner and requires no changes to existing
>>>> pygobject code. The solution is to link __init__.py from overrides
>>>> modules to system pygobject overrides directory in case overrides
>>>> module is installed outside of pygobject tree. With autotools, this
>>>> can be done easily by using install-exec-hook -
>>>> gst-python-symlink-__init__.py-to-system-gi-overrides-on-install.patch
>>>> demonstrates this idea on gst-python.
>>>>
>>>> Cheers,
>>>> Lukas Vacek
>>>>
>>>> On Mon, Oct 22, 2012 at 1:35 PM, Lukas Vacek <lucas vacek gmail com> wrote:
>>>>> Hi All,
>>>>>
>>>>> I have found out it's not possible to load installed GI overrides from
>>>>> a different directory than where PyGObject itself is installed
>>>>> (/usr/lib/python2.7/dist-packages/ in my case).
>>>>>
>>>>> Ideally gi overrides modules could be installed to user site-packages
>>>>> directory and still would be loaded by system installation of
>>>>> PyGObject.
>>>>>
>>>>> The problem is that the overrides module can't install __init__.py
>>>>> file because it would conflict with __init__.py already installed by
>>>>> system PyGObject. (Or even if the file was not installed in the same
>>>>> location but the installed overrides module (with own __init__.py)
>>>>> would appear before system PyGObject in sys.path it would be loaded
>>>>> instead of PyGObject as gi module and break things. )
>>>>>
>>>>> However, __init__.py must be present in the gi overrides module
>>>>> directory in order to be recognized by __import__ call made by
>>>>> PyGObject when loading overrides module (module.py around line 244).
>>>>>
>>>>> This makes it complicated to use gi overrides module from a different
>>>>> directory. The workaround is to load _uninstalled_ gi overrides
>>>>> modules with PYTHONPATH which explicitly mentions PyGObject path
>>>>> before the path to overrides modules. For example, this is how I run
>>>>> pitivi: PYTHONPATH=/usr/lib/python2.7/dist-packages/:~/dev/gst-python/
>>>>> python bin/pitivi
>>>>>
>>>>> Pitivi bugs https://bugzilla.gnome.org/show_bug.cgi?id=686341 and
>>>>> https://bugzilla.gnome.org/show_bug.cgi?id=686342 seem to be related
>>>>> to/caused by this.
>>>>>
>>>>> To solve this pygobject might look for a ".py" file instead of a module.
>>>>>
>>>>> Thanks!
>>>>> Lukas Vacek
>>>>
>>>> _______________________________________________
>>>> python-hackers-list mailing list
>>>> python-hackers-list gnome org
>>>> https://mail.gnome.org/mailman/listinfo/python-hackers-list
>>>>