Upstream change in Python breaks genshi test suite

[robie.basak@…]

The genshi test suite gets stuck in an infinite loop when run against the latest upstream 2.7 branch.

Python issue ​http://bugs.python.org/issue16152 now has a fix that is committed to the 2.7 branch. This changes the definition of tokenize.PseudoToken. This constant is used in genshi.template.interpolation. The change to this constant causes the test genshi.template.tests.interpolation.py InterpolateTestCase.test_interpolate_full_mismatched_brackets to end up in an infinite loop.

A test case that demonstrates the problem is below. It fails on a build of Python 2.7 from the Python upstream 2.7 hg branch.

The current Ubuntu development release is based on the Python upstream 2.7 hg branch, so this causes genshi to fail to build on Ubuntu.

This affects both your 0.6 release and your svn trunk (r1209).

A workaround is to hardcode a local definition of PseudoToken as it was in previous Python versions. But it isn't clear to me if this is an acceptable fix. Please could you provide some guidance on what should be going on here, and how we might patch Ubuntu so that we can build genshi?

Steps to reproduce:

1. Install Python 2.7 from Python upstream hg branch 2.7, or use the current Ubuntu development release (Raring).
2. python setup.py build
3. python setup.py test

Expected result: test suite should complete

Actual result: test suite hangs on test_interpolate_full_mismatched_brackets

Test case:

#!/usr/bin/python

import re
import unittest

from tokenize import (
    Whitespace, Number, Funny, ContStr, Name, Comment, Triple, group
)

token_re = re.compile('%s|%s(?s)' % (
    r'[uU]?[rR]?("""|\'\'\')((?<!\\)\\\1|.)*?\1',
    # PseudoToken =
    Whitespace + group(
        group(
            # PseudoExtras =
            # r'\\\r?\n',  # with this (< 2.7.4) it works
            r'\\\r?\n|\Z', # (>= 2.7.4) 
            Comment, Triple),
        Number, Funny, ContStr, Name)
    ))

class TestCase(unittest.TestCase):
    def test_regexp_does_not_match(self):
        self.assertEqual(token_re.match('${{1:2}', 7), None)

if __name__ == '__main__':
    unittest.main()

[robie.basak@…]

Ubuntu bug: ​https://bugs.launchpad.net/ubuntu/+source/genshi/+bug/1097776

[cboos]

I've looked up a bit the ongoing discussion on Launchpad.

IIUC, it remains to be verified if after the change discussed in ​https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1097783/comments/2, the problem persists or not.

[anonymous]

Python in Ubuntu raring is now updated to fix the issue in ​https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1097783, so just this issue is left. I've just tested it using 0.6, and the problem remains.

Do you think this should be fixed by using the same definition of PseudoToken as existed in previous versions of Python, or in some other way? If we patch genshi in Ubuntu to use the old definition of PseudoToken instead of importing from tokenize, will this cause any problems in genshi's behaviour?

[cboos]

I updated your test case with an expanded version, so that we can see the failure with released Python versions.

So first, when you say "hangs", I think you mean there's a test failure, which is much much less scary than a real "hang" of the process ;-)

I'm not exactly aware of the implementation details in Genshi, but to me it seems that such a regexp shouldn't match for ${{...}, so we can't use the newer PseudoToken and we should probably recompose the older one.

[hodgestar]

The infinite loop is a result of PseudoToken now possibly matching strings of length zero at the end of the string. The patch to CPython 2.7 adds a check to its tokenizing loop for this case. I added an equivalent check to Genshi in trunk in r1213 and backported it to 0.6.x in r1214 and 0.7.x in r1215.