Home » Coding » Python » weakrefs and bound methods
| weakrefs and bound methods [message #15701] |
Sun, 07 October 2007 10:51  |
 Mathias Panzenboeck
Messages: 5
Registered: October 2007 |
Junior Member |
|
|
Hi.
I have a problem with weak refs and bound methods. The best explanation for the
problem is a short bit of code. I have the three classes Wrapper, Foo and Bar:
import weakref
class Wrapper(object):
def __init__(self,x):
self.x = weakref.ref(x)
def __call__(self,*args,**kwargs):
x = self.x()
if x is None:
print "lost reference"
else:
return x(*args,**kwargs)
class Foo(object):
def __init__(self):
self._methods = set()
self._methods.add(Wrapper(self._foo))
def _foo(self):
print "_foo"
def callMethods(self):
for method in self._methods:
method()
def __del__(self):
print "del Foo"
class Bar(object):
def __init__(self):
self._methods = set()
self._methods.add(self._foo)
def _foo(self):
print "_foo"
def callMethods(self):
for method in self._methods:
method()
def __del__(self):
print "del Bar"
Now look what happens when I do this:
>>> f=Foo()
>>> f.callMethods()
lost reference
>>> del f
del Foo
>>> b=Bar()
>>> b.callMethods()
_foo
>>> del b
>>>
Foo looses the reference to its method but Bar on the other hand has a refloop and
never gets deleted.
Does anyone know what happens here? How can I write such classes without refloop and
without lost reference?
I'm using:
Python 2.5.1 (r251:54863, May 2 2007, 16:56:35)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
-panzi
|
|
| | | | | |
| Re: weakrefs and bound methods [message #15766 is a reply to message #15701 ] |
Sun, 07 October 2007 13:14  |
steve
Messages: 106
Registered: July 2007 |
Senior Member |
|
|
On Sun, 07 Oct 2007 16:38:23 +0000, Michele Simionato wrote:
> On Oct 7, 12:26 pm, Marc 'BlackJack' Rintsch <bj_...@gmx.net> wrote:
>
>> Drop all those `__del__()` methods as they prevent the garbage
>> collector from collecting "cycles".
>
> I fully agree and I will add that __del__ methods are always a bad idea.
Always?
I recently wrote a bit of code where I needed to check that releasing the
first object in a tree-like structure would allow Python to garbage
collect all the other objects in a tree. I thought it would, but I wanted
to be sure ("don't guess, test"), so I wrote a simple class, gave it a
__del__ method that just printed self, inserted them in the tree, and
then deleted the first one.
Worked like a charm.
Without __del__, what should I have done to test that my code was
deleting objects and not leaking memory?
What should I do when my objects need to perform some special processing
when they are freed, if I shouldn't use __del__?
--
Steven.
|
|
|
| Re: weakrefs and bound methods [message #15768 is a reply to message #15762 ] |
Sun, 07 October 2007 13:25 |
Steven Bethard
Messages: 42
Registered: August 2007 |
Member |
|
|
Mathias Panzenboeck wrote:
> Marc 'BlackJack' Rintsch wrote:
>> ``del b`` just deletes the name `b`. It does not delete the object.
>> There's still the name `_` bound to it in the interactive interpreter.
>> `_` stays bound to the last non-`None` result in the interpreter.
>
> Actually I have the opposite problem. The reference (to the bound method)
> gets lost but it shouldn't!
Ahh, so you expected that ``Wrapper(self._foo)`` would not immediately
lose the reference? It will, because every time you write
``self._foo``, a new bound method is created::
>>> class C(object):
... def foo(self):
... pass
...
>>> f = C.foo
>>> g = C.foo
>>> id(f), id(g)
(14931448, 14891008)
Thus, there is only the one reference to the bound method, and by
wrapping it in a weakref, you are allowing it to disappear immediately::
>>> x = weakref.ref(C.foo)
>>> print x()
None
What behavior do you want here? That is, when were you hoping that the
bound method would disappear?
STeVe
|
|
|
| Re: weakrefs and bound methods [message #15775 is a reply to message #15766 ] |
Sun, 07 October 2007 13:34 |
Michele Simionato
Messages: 60
Registered: August 2007 |
Member |
|
|
On Oct 7, 1:14 pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.au> wrote:
> On Sun, 07 Oct 2007 16:38:23 +0000, Michele Simionato wrote:
> > On Oct 7, 12:26 pm, Marc 'BlackJack' Rintsch <bj_...@gmx.net> wrote:
>
> >> Drop all those `__del__()` methods as they prevent the garbage
> >> collector from collecting "cycles".
>
> > I fully agree and I will add that __del__ methods are always a bad idea.
>
> Always?
>
> I recently wrote a bit of code where I needed to check that releasing the
> first object in a tree-like structure would allow Python to garbage
> collect all the other objects in a tree. I thought it would, but I wanted
> to be sure ("don't guess, test"), so I wrote a simple class, gave it a
> __del__ method that just printed self, inserted them in the tree, and
> then deleted the first one.
>
> Worked like a charm.
>
> Without __del__, what should I have done to test that my code was
> deleting objects and not leaking memory?
>
> What should I do when my objects need to perform some special processing
> when they are freed, if I shouldn't use __del__?
The best thing is to use explicit resource management,
for instance with a try .. finally or with the "with" statement
in Python 2.5. The next best thing is to use weak references.
I have some code for various experiments with wearefs I did
some time ago, here it is:
import itertools, weakref, sys, gc
reference_list = [] # cannot be a set, you would lose references
resource_counter = itertools.count(1)
def resource(before_closing_callback=None,
after_closing_callback=None):
private = '_resource_%s' % resource_counter.next()
def get(self):
return getattr(self, private)
def set(self, resource):
setattr(self, private, resource)
def close(ref):
if before_closing_callback:
before_closing_callback(resource)
resource.close()
if after_closing_callback:
after_closing_callback(resource)
reference_list.remove(ref)
reference_list.append(weakref.ref(self, close))
return property(get, set)
class FakeResource(object):
def __init__(self, name):
print 'opening resource %s' % name
self.name = name
def close(self):
print 'closing resource %s' % self.name
def __repr__(self):
return '<FakeResource %r>' % self.name
class Example(object):
def __init__(self):
self.resource1 = FakeResource('r1')
self.resource2 = FakeResource('r2')
def __del__(self):
print '**************'
self.resource1.close()
self.resource2.close()
def warn_before_closing(res):
sys.stdout.write('going to close %s\n' % res)
class Example2(object):
resource1 = resource(warn_before_closing)
resource2 = resource()
def __init__(self):
self.resource1 = FakeResource('r1')
self.resource2 = FakeResource('r2')
gc.set_debug(gc.DEBUG_LEAK)
#e = Example()
e = Example2()
e.e = e
del e
print reference_list
|
|
|
| Re: weakrefs and bound methods [message #17427 is a reply to message #15766 ] |
Sun, 07 October 2007 15:55 |
aleax
Messages: 95
Registered: July 2007 |
Member |
|
|
Steven D'Aprano <steve@REMOVE-THIS-cybersource.com.au> wrote:
...
> Without __del__, what should I have done to test that my code was
> deleting objects and not leaking memory?
See module gc in the Python standard library.
> What should I do when my objects need to perform some special processing
> when they are freed, if I shouldn't use __del__?
The solid, reliable way is:
from __future__ import with_statement
and use module contextlib from the Python standard library (or handcode
an __exit__ method, but that's rarely needed), generating these special
objects that require special processing only in 'with' statements. This
"resource acquisition is initialization" (RAII) pattern is the RIGHT way
to ensure timely finalization (particularly but not exclusively in
garbage-collected languages, and particularly but not exclusively to
ease portability to different garbage collection strategies -- e.g.,
among CPython and future versions of IronPython and/or Jython that will
support the with statement).
An alternative that will work in pre-2.5 Python (and, I believe but I'm
not sure, in Jython and IronPython _today_) is to rely on the weakref
module of the standard Python library. If your finalizer, in order to
perform "special processing", requires access to some values that depend
on the just-freed object, you'll have to carefully stash those values
"elsewhere", because the finalizer gets called _after_ the object is
freed (this crucial bit of sequencing semantics is what allows weak
references to work while "strong finalizers" [aka destructors] don't
play well with garbage collection when reference-loops are possible).
E.g., weakref.ref instances are hashable, so you can keep a per-class
dict keyed by them to hold the special values that are needed for
special processing at finalization, and use accessors as needed to make
those special values still look like attributes of instances of your
class.
E.g., consider:
import weakref
class ClosingAtDel(object):
_xs = {}
def __init__(self, x):
self._r = weakref.ref(self, self._closeit)
self._xs[self._r] = x
@property
def x(self):
return self._xs[self._r]
@classmethod
def _closeit(cls, theweakref):
cls._xs[theweakref].close()
del cls._xs[theweakref]
This will ensure that .close() is called on the object 'wrapped' in the
instance of ClosingAtDel when the latter instance goes away -- even when
the "going away" is due to a reference loop getting collected by gc. If
ClosingAtDel had a __del__ method, that would interfere with the garbage
collection. For example, consider adding to that class the following
test/example code:
class Zap(object):
def close(self): print 'closed', self
c = ClosingAtDel(Zap())
d = ClosingAtDel(Zap())
print c.x, d.x
# create a reference loop
c.xx = d; d.xx = c
# garbage-collect it anyway
import gc
del c; del d; gc.collect()
print 'done!'
you'll get a pleasant, expected output:
$ python wr.py
<__main__.Zap object at 0x6b430> <__main__.Zap object at 0x6b490>
closed <__main__.Zap object at 0x6b430>
closed <__main__.Zap object at 0x6b490>
done!
Suppose that ClosingAtDel was instead miscoded with a __del__, e.g.:
class ClosingAtDel(object):
def __init__(self, x):
self.x = x
def __del__(self):
self.x.close()
Now, the same test/example code would emit a desolating...:
$ python wr.py
<__main__.Zap object at 0x6b5b0> <__main__.Zap object at 0x6b610>
done!
I.e., the assumed-to-be-crucial calls to .close() have NOT been
performed, because __del__ inhibits collection of reference-looping
garbage. _Ensuring_ you always avoid reference loops (in intricate
real-life cases) is basically unfeasible (that's why we HAVE gc in the
first place -- for non-loopy cases, reference counting suffices;-), so
the best strategy is to avoid coding __del__ methods, just as Marc
recommends.
Alex
|
|
| | | |
| Re: weakrefs and bound methods [message #17455 is a reply to message #15701 ] |
Sun, 07 October 2007 23:28 |
steven
Messages: 5
Registered: September 2007 |
Junior Member |
|
|
On Sun, 07 Oct 2007 12:55:29 -0700, Alex Martelli wrote:
>> What should I do when my objects need to perform some special
>> processing when they are freed, if I shouldn't use __del__?
>
> The solid, reliable way is:
>
> from __future__ import with_statement
>
> and use module contextlib from the Python standard library (or handcode
> an __exit__ method, but that's rarely needed), generating these special
> objects that require special processing only in 'with' statements. This
> "resource acquisition is initialization" (RAII) pattern is the RIGHT way
> to ensure timely finalization (particularly but not exclusively in
> garbage-collected languages, and particularly but not exclusively to
> ease portability to different garbage collection strategies -- e.g.,
> among CPython and future versions of IronPython and/or Jython that will
> support the with statement).
Hmmm... I'm not sure I understand how a with statement is meant to
replace class.__del__ in practice. It seems to me that the two things
have different uses. with is useful when you create an object, do
something with it, dispose of it in an orderly fashion, and then
continue. How does that help when you create an unknown number of long-
lasting objects where you can't predict how and when they will be
disposed of?
Here's a minimal example. I have a module that returns parrot objects:
# Parrot module
class Parrot(object):
def __del__(self):
print "I'm now pushing up the daisies"
# Callers code:
import Parrot
p = Parrot.Parrot()
alist = [Parrot.Parrot() for i in range(1000)]
# do something with p and alist
print p, alist
# instances are deleted in arbitrary order
del p
del alist[3]
del alist[853]
del alist[701]
del alist
It seems to me that you are saying to re-write the above as something
like this:
# Parrot module
class Parrot(object):
pass
class killed(object): # context manager
def __exit__(self, *exc_info):
print "I'm now pushing up the daisies"
# Callers code:
import Parrot
with killed(Parrot.Parrot()) as p:
# ... er, what do I do with alist???
# do arbitrary stuff with p and alist
print p, alist
Now, I don't think the above is feasible. What am I missing?
> An alternative that will work in pre-2.5 Python (and, I believe but I'm
> not sure, in Jython and IronPython _today_) is to rely on the weakref
> module of the standard Python library.
....
I'm going to need some time to play around with that, but it seems
reasonably straightforward. But the obvious thing I can see is that
there's an awful lot of boilerplate code that needs to be written for
each and every class that needs a "weak finalizer".
That is to say, Python "guarantees" that __del__ will always be called
(unless it isn't... "guarantee void on planet Earth") for your objects,
so the developer doesn't need to do anything except write the finalizer.
To do the right thing with weakrefs means that not only does the
developer have to write the finalizer, but he has to write the code to
ensure the finalizer is called. I'm not sure that's a step forward.
--
Steven.
|
|
|
| Re: weakrefs and bound methods [message #17457 is a reply to message #17455 ] |
Mon, 08 October 2007 00:06 |
Michele Simionato
Messages: 60
Registered: August 2007 |
Member |
|
|
On Oct 7, 11:28 pm, Steven D'Aprano
<ste...@REMOVE.THIS.cybersource.com.au> wrote:
> On Sun, 07 Oct 2007 12:55:29 -0700, Alex Martelli wrote:
> >> What should I do when my objects need to perform some special
> >> processing when they are freed, if I shouldn't use __del__?
>
> > The solid, reliable way is:
>
> > from __future__ import with_statement
>
> > and use module contextlib from the Python standard library (or handcode
> > an __exit__ method, but that's rarely needed), generating these special
> > objects that require special processing only in 'with' statements. This
> > "resource acquisition is initialization" (RAII) pattern is the RIGHT way
> > to ensure timely finalization (particularly but not exclusively in
> > garbage-collected languages, and particularly but not exclusively to
> > ease portability to different garbage collection strategies -- e.g.,
> > among CPython and future versions of IronPython and/or Jython that will
> > support the with statement).
>
> Hmmm... I'm not sure I understand how a with statement is meant to
> replace class.__del__ in practice. It seems to me that the two things
> have different uses. with is useful when you create an object, do
> something with it, dispose of it in an orderly fashion, and then
> continue. How does that help when you create an unknown number of long-
> lasting objects where you can't predict how and when they will be
> disposed of?
>
> Here's a minimal example. I have a module that returns parrot objects:
>
> # Parrot module
> class Parrot(object):
> def __del__(self):
> print "I'm now pushing up the daisies"
>
> # Callers code:
> import Parrot
> p = Parrot.Parrot()
> alist = [Parrot.Parrot() for i in range(1000)]
> # do something with p and alist
> print p, alist
> # instances are deleted in arbitrary order
> del p
> del alist[3]
> del alist[853]
> del alist[701]
> del alist
>
> It seems to me that you are saying to re-write the above as something
> like this:
>
> # Parrot module
> class Parrot(object):
> pass
>
> class killed(object): # context manager
> def __exit__(self, *exc_info):
> print "I'm now pushing up the daisies"
>
> # Callers code:
> import Parrot
> with killed(Parrot.Parrot()) as p:
> # ... er, what do I do with alist???
> # do arbitrary stuff with p and alist
> print p, alist
>
> Now, I don't think the above is feasible. What am I missing?
Rename your __del__ to close and call it explicitely:
p.close()
alist[3].close()
alist[853].close()
alist[701].close()
> > An alternative that will work in pre-2.5 Python (and, I believe but I'm
> > not sure, in Jython and IronPython _today_) is to rely on the weakref
> > module of the standard Python library.
>
> ...
>
> I'm going to need some time to play around with that, but it seems
> reasonably straightforward. But the obvious thing I can see is that
> there's an awful lot of boilerplate code that needs to be written for
> each and every class that needs a "weak finalizer".
>
> That is to say, Python "guarantees" that __del__ will always be called
> (unless it isn't... "guarantee void on planet Earth") for your objects,
> so the developer doesn't need to do anything except write the finalizer.
Yep, the problem is that __del__ does NOT guarantee anything.
So, it *looks* easy, but it is actually very fragile and in
subtle cases can betray you without telling. I, for one,
I don't like to have bombs in my code waiting to explode
at some unknown moment ...
> To do the right thing with weakrefs means that not only does the
> developer have to write the finalizer, but he has to write the code to
> ensure the finalizer is called. I'm not sure that's a step forward.
Yep, we would need a better support for finalizers in the
standard library. For the moment, there are various recipes
in the Python cookbook. See also this post:
http://groups.google.com/group/comp.lang.python/browse_frm/t hread/b09e02b7c67c8a3b/1ef0f64bb539cfb0?hl=en&lnk=gst&am p;q=simionato+__del__&rnum=1#1ef0f64bb539cfb0
|
|
|
Goto Forum:
Current Time: Fri May 16 02:42:37 EDT 2008
Total time taken to generate the page: 0.14158 seconds |
Members
Search
Help
Register
Login
Home
