Edgewall Software

Opened 10 years ago

Closed 10 years ago

#540 closed defect (fixed)

Upstream change in Python breaks genshi test suite

Reported by: robie.basak@… Owned by: cmlenz
Priority: major Milestone: 0.7
Component: General Version: 0.6
Keywords: Cc:

Description (last modified by cboos)

The genshi test suite gets stuck in an infinite loop when run against the latest upstream 2.7 branch.

Python issue http://bugs.python.org/issue16152 now has a fix that is committed to the 2.7 branch. This changes the definition of tokenize.PseudoToken. This constant is used in genshi.template.interpolation. The change to this constant causes the test genshi.template.tests.interpolation.py InterpolateTestCase.test_interpolate_full_mismatched_brackets to end up in an infinite loop.

A test case that demonstrates the problem is below. It fails on a build of Python 2.7 from the Python upstream 2.7 hg branch.

The current Ubuntu development release is based on the Python upstream 2.7 hg branch, so this causes genshi to fail to build on Ubuntu.

This affects both your 0.6 release and your svn trunk (r1209).

A workaround is to hardcode a local definition of PseudoToken as it was in previous Python versions. But it isn't clear to me if this is an acceptable fix. Please could you provide some guidance on what should be going on here, and how we might patch Ubuntu so that we can build genshi?

Steps to reproduce:

  1. Install Python 2.7 from Python upstream hg branch 2.7, or use the current Ubuntu development release (Raring).
  2. python setup.py build
  3. python setup.py test

Expected result: test suite should complete

Actual result: test suite hangs on test_interpolate_full_mismatched_brackets

Test case:


import re
import unittest

from tokenize import (
    Whitespace, Number, Funny, ContStr, Name, Comment, Triple, group

token_re = re.compile('%s|%s(?s)' % (
    # PseudoToken =
    Whitespace + group(
            # PseudoExtras =
            # r'\\\r?\n',  # with this (< 2.7.4) it works
            r'\\\r?\n|\Z', # (>= 2.7.4) 
            Comment, Triple),
        Number, Funny, ContStr, Name)

class TestCase(unittest.TestCase):
    def test_regexp_does_not_match(self):
        self.assertEqual(token_re.match('${{1:2}', 7), None)

if __name__ == '__main__':

Change History (5)

comment:2 Changed 10 years ago by cboos

I've looked up a bit the ongoing discussion on Launchpad.

IIUC, it remains to be verified if after the change discussed in https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1097783/comments/2, the problem persists or not.

comment:3 Changed 10 years ago by anonymous

Python in Ubuntu raring is now updated to fix the issue in https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1097783, so just this issue is left. I've just tested it using 0.6, and the problem remains.

Do you think this should be fixed by using the same definition of PseudoToken as existed in previous versions of Python, or in some other way? If we patch genshi in Ubuntu to use the old definition of PseudoToken instead of importing from tokenize, will this cause any problems in genshi's behaviour?

comment:4 Changed 10 years ago by cboos

  • Description modified (diff)

I updated your test case with an expanded version, so that we can see the failure with released Python versions.

So first, when you say "hangs", I think you mean there's a test failure, which is much much less scary than a real "hang" of the process ;-)

I'm not exactly aware of the implementation details in Genshi, but to me it seems that such a regexp shouldn't match for ${{...}, so we can't use the newer PseudoToken and we should probably recompose the older one.

comment:5 Changed 10 years ago by hodgestar

  • Resolution set to fixed
  • Status changed from new to closed

The infinite loop is a result of PseudoToken now possibly matching strings of length zero at the end of the string. The patch to CPython 2.7 adds a check to its tokenizing loop for this case. I added an equivalent check to Genshi in trunk in r1213 and backported it to 0.6.x in r1214 and 0.7.x in r1215.

Note: See TracTickets for help on using tickets.