Ticket #190 (closed defect: fixed)
Memory leak when Python code in templates raises an exception
| Reported by: | cboos | Owned by: | cmlenz |
|---|---|---|---|
| Priority: | critical | Milestone: | 0.5 |
| Component: | Expression evaluation | Version: | 0.4.4 |
| Keywords: | memory | Cc: |
Description
This is a follow-up to the some of the memory issues we've seen in Trac (#T6614). athomas was able to create a sample program that reproduces the issue:
import gc from itertools import groupby from genshi.template import MarkupTemplate from genshi.template.eval import UndefinedError m = MarkupTemplate(""" <html xmlns:py="http://genshi.edgewall.org/"> <body> <py:for each="i, j in groupby(test, key=lambda i: len(i))"> ${say.hello} </py:for> </body> </html> """, lookup='lenient') while True: test = dict.fromkeys(map(str, range(1000, 100000))) try: print m.generate(test=test, groupby=groupby) except UndefinedError: pass print gc.collect() print len(gc.get_objects())
Running it shows something like:
$ python25 aat2.py 354 12926 0 12949 0 12972 0 12995 0 13018 0 13041 0 13064 0 ...
And it won't stop growing. This is with latest Genshi (r800).
Now, the interesting thing is that when we revert r770, the problem seems to disappear:
$ python25 aat2.py 354 12924 16 12924 16 12924 16 12924 16 12924 16 12924 16 12924 16 12924 16 ...
and memory usage is constant as well.
To me, these results seem to indicate clearly that the following eval seems to hold a hidden reference to the data when an exception is raised.
def evaluate(self, data): """Evaluate the expression against the given data dictionary. :param data: a mapping containing the data to evaluate against :return: the result of the evaluation """ __traceback_hide__ = 'before_and_this' _globals = self._globals() _globals['__data__'] = data return eval(self.code, _globals, {'__data__': data})
(see source:trunk/genshi/template/eval.py@800#L135)
When removing the locals from that last line:
-
genshi/template/eval.py
141 141 __traceback_hide__ = 'before_and_this' 142 142 _globals = self._globals() 143 143 _globals['__data__'] = data 144 return eval(self.code, _globals , {'__data__': data})144 return eval(self.code, _globals) 145 145 146 146 147 147 class Suite(Code):
the problem persists, so that reference seems to be related to the globals (and by the way, this makes me think of that blog entry "How to Add Memory Leaks to Python", for which I never saw an explanation, maybe it's the same underlying bug).
