Memory leak in _speedups.c

[cboos]

While tracking down the memory leak we have in Trac 0.11 [t6303], I had a go at removing the _speedups.so library and that effectively seems to be enough to stabilize the memory usage.

Steps to repeat

Use tracd 0.11 with latest Genshi trunk on a t.e.o clone, then repeatedly query: http://localhost:8910/env/timeline?from=2007-08-20T10%3A15%3A47Z%2B0200&precision=second

• Without _speedups.so, the memory usage stabilizes at 135 Mbytes.
• With _speedups.so, each request adds between 500 and 800K.

In addition, I'm doing a gc.collect() at the end of each request:

trac/web/main.py

@@ -382 +382 @@
     finally:
         if env and not run_once:
             env.shutdown(threading._get_ident())
+            import gc
+            print '----' * 10
+            gc.set_debug(gc.DEBUG_UNCOLLECTABLE)
+            print 'Garbage Collection', gc.collect()
+            print 'Garbage left:', repr(gc.garbage)
+            print 'Objects tracked:', repr(len(gc.get_objects()))
 
 def _dispatch_request(req, env, env_error):
     resp = []

The usual output is:

Garbage Collection 0
Garbage left: []
Objects tracked: 47570

(and it stays at 47570)

Reasons

I've not yet located the problem, but so far escape() seems suspicious: the in unicode string doesn't get DECREFed in the inn != 0 case.

Trying the following:

genshi/_speedups.c

@@ -87 +87 @@
 
     out = (PyUnicodeObject*) PyUnicode_FromUnicode(NULL, len);
     if (out == NULL) {
+        Py_DECREF((PyObject *) in);
         return NULL;
     }
 
@@ -130 +131 @@
         inp++;
     }
 
+    Py_DECREF((PyObject *) in);
+
     args = PyTuple_New(1);
     if (args == NULL) {
         Py_DECREF((PyObject *) out);

which seems better for that particular function, doesn't seem to impact significantly on the leak, so there must be other things.

[cboos]

Fix some more issues. The Py_DECREF(unicode); near line 380 seems important. Things seems to be a bit better with that patch, approx 400K increase for each request.

[cboos]

Updated version: this one fixes the leak. The major culprit appeared to be a missing DECREF for the PySequence?_GetItem in Markup_join, near line 269.

[cmlenz]

Applied in [781]. Thanks!

[hyuga <hyugaricdeau@…>]

Reopening on the suggestion of cboos on the mailing list. The only issue at this point was a missing Py_DECREF(format) at the end of Markup_repr().

[hyuga <hyugaricdeau@…>]

Patch for the missing PyDECREF. Other than that, I didn't notice anything.

[cmlenz]

Applied in [797]. Thanks!