Edgewall Software

Ticket #540 (closed defect: fixed)

Opened 22 months ago

Last modified 21 months ago

Upstream change in Python breaks genshi test suite

Reported by: robie.basak@… Owned by: cmlenz
Priority: major Milestone: 0.7
Component: General Version: 0.6
Keywords: Cc:

Description (last modified by cboos) (diff)

The genshi test suite gets stuck in an infinite loop when run against the latest upstream 2.7 branch.

Python issue  http://bugs.python.org/issue16152 now has a fix that is committed to the 2.7 branch. This changes the definition of tokenize.PseudoToken. This constant is used in genshi.template.interpolation. The change to this constant causes the test genshi.template.tests.interpolation.py InterpolateTestCase.test_interpolate_full_mismatched_brackets to end up in an infinite loop.

A test case that demonstrates the problem is below. It fails on a build of Python 2.7 from the Python upstream 2.7 hg branch.

The current Ubuntu development release is based on the Python upstream 2.7 hg branch, so this causes genshi to fail to build on Ubuntu.

This affects both your 0.6 release and your svn trunk (r1209).

A workaround is to hardcode a local definition of PseudoToken as it was in previous Python versions. But it isn't clear to me if this is an acceptable fix. Please could you provide some guidance on what should be going on here, and how we might patch Ubuntu so that we can build genshi?

Steps to reproduce:

  1. Install Python 2.7 from Python upstream hg branch 2.7, or use the current Ubuntu development release (Raring).
  2. python setup.py build
  3. python setup.py test

Expected result: test suite should complete

Actual result: test suite hangs on test_interpolate_full_mismatched_brackets

Test case:


import re
import unittest

from tokenize import (
    Whitespace, Number, Funny, ContStr, Name, Comment, Triple, group

token_re = re.compile('%s|%s(?s)' % (
    # PseudoToken =
    Whitespace + group(
            # PseudoExtras =
            # r'\\\r?\n',  # with this (< 2.7.4) it works
            r'\\\r?\n|\Z', # (>= 2.7.4) 
            Comment, Triple),
        Number, Funny, ContStr, Name)

class TestCase(unittest.TestCase):
    def test_regexp_does_not_match(self):
        self.assertEqual(token_re.match('${{1:2}', 7), None)

if __name__ == '__main__':


Change History

Changed 22 months ago by cboos

I've looked up a bit the ongoing discussion on Launchpad.

IIUC, it remains to be verified if after the change discussed in  https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1097783/comments/2, the problem persists or not.

Changed 21 months ago by anonymous

Python in Ubuntu raring is now updated to fix the issue in  https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1097783, so just this issue is left. I've just tested it using 0.6, and the problem remains.

Do you think this should be fixed by using the same definition of PseudoToken as existed in previous versions of Python, or in some other way? If we patch genshi in Ubuntu to use the old definition of PseudoToken instead of importing from tokenize, will this cause any problems in genshi's behaviour?

Changed 21 months ago by cboos

  • description modified (diff)

I updated your test case with an expanded version, so that we can see the failure with released Python versions.

So first, when you say "hangs", I think you mean there's a test failure, which is much much less scary than a real "hang" of the process ;-)

I'm not exactly aware of the implementation details in Genshi, but to me it seems that such a regexp shouldn't match for ${{...}, so we can't use the newer PseudoToken and we should probably recompose the older one.

Changed 21 months ago by hodgestar

  • status changed from new to closed
  • resolution set to fixed

The infinite loop is a result of PseudoToken now possibly matching strings of length zero at the end of the string. The patch to CPython 2.7 adds a check to its tokenizing loop for this case. I added an equivalent check to Genshi in trunk in r1213 and backported it to 0.6.x in r1214 and 0.7.x in r1215.

Add/Change #540 (Upstream change in Python breaks genshi test suite)


E-mail address and user name can be saved in the Preferences.

Change Properties
<Author field>
as closed
The resolution will be deleted. Next status will be 'reopened'
Note: See TracTickets for help on using tickets.