Hi, all. I haven''t felt at liberty to go axing test cases, but some of the 33 that are failing now are ones I don''t think we want to pass? cruft coming from Markdown or made up to fill a gap some years ago. I''d like to float the proposition that we make Super (and thus RedCloth 4) behave as identically as possible to Textile2 as working on http://textile.thresholdstate.com/ (the textpattern site seems to be running an older version of this). There are some new things Textile2 does that SRC does not and I''d like to impliment those and also strip out the cruft I mentioned. This may break backwards compatibility for some people''s content (I''m thinking perhaps the poignant test cases), but I think that''s okay with a major release (3 -> 4) such as this would be. You can''t ever move ahead if you don''t own up to mistakes and leave them behind. What do you all think? Can I start cleaning up test cases to be how they ought to be? I''ll do my research carefully and be testing it with real content (in a university CMS) over the next few months. I want the end result to be as stable, predictable, and intuitive as anyone. Thanks! Jason Garber -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/redcloth-upwards/attachments/20070823/974f1689/attachment.html
>Hi, all. I haven''t felt at liberty to go axing test cases, but some >of the 33 that are failing now are ones I don''t think we want to >pass-cruft coming from Markdown or made up to fill a gap some years >ago. > >I''d like to float the proposition that we make Super (and thus >RedCloth 4) behave as identically as possible to Textile2 as working >on ><http://textile.thresholdstate.com>http://textile.thresholdstate.com/ >(the textpattern site seems to be running an older version of this). >There are some new things Textile2 does that SRC does not and I''d >like to impliment those and also strip out the cruft I mentioned. >This may break backwards compatibility for some people''s content >(I''m thinking perhaps the poignant test cases),Can you categorize the way the sets of functionality relate: I''m most interested in the sets in this order: * functionality in existing SRC that is incompatible with T2. By this I mean that just because this functionality exists SRC could not in principle pass a complete suite of T2 tests. * functionality in T2 that is NOT in existing SRC * functionality in existing SRC that not in T2. Is there a document that clearly specifies the language rules/spec of T2? In general I think it''s great that you want to clean up the test cases. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/redcloth-upwards/attachments/20070823/ccaa81fc/attachment.html
Unfortunately, I can''t come up with a list of the sets you request, Stephen. Off-hand, I know definition lists are done differently in RedCloth (not implemented in SRC) than in T2; backticks to denote code were in RC (don''t know about SRC). What I''d like to do is hook up another test that runs all the cases (yml files) through T2 (php) and compares the output. I don''t necessarily want that to be a permanent part of SRC''s test suite, but it would be interesting to see exactly how they handle things differently. I think I''ve encountered a few cases where T2 has undesirable output but SRC does it right. Obviously we''d want to preserve SRC''s way. Language ref? The Movable Type Textile2 manual is the best I''ve found. http://www.bradchoate.com/mt/docs/mtmanual_textile2.html Certainly wish Textile2 (php) had some test cases that I could just port over to Ruby! On Aug 23, 2007, at 3:48 PM, Stephen Bannasch wrote:>> Hi, all. I haven''t felt at liberty to go axing test cases, but >> some of the 33 that are failing now are ones I don''t think we want >> to pass-cruft coming from Markdown or made up to fill a gap some >> years ago. >> >> I''d like to float the proposition that we make Super (and thus >> RedCloth 4) behave as identically as possible to Textile2 as >> working on http://textile.thresholdstate.com/ (the textpattern >> site seems to be running an older version of this). There are some >> new things Textile2 does that SRC does not and I''d like to >> impliment those and also strip out the cruft I mentioned. This may >> break backwards compatibility for some people''s content (I''m >> thinking perhaps the poignant test cases), > > Can you categorize the way the sets of functionality relate: > > I''m most interested in the sets in this order: > > * functionality in existing SRC that is incompatible with T2. By > this I mean that just because this functionality exists SRC could > not in principle pass a complete suite of T2 tests. > * functionality in T2 that is NOT in existing SRC > * functionality in existing SRC that not in T2. > > Is there a document that clearly specifies the language rules/spec > of T2? > > In general I think it''s great that you want to clean up the test > cases. > _______________________________________________ > Redcloth-upwards mailing list > Redcloth-upwards at rubyforge.org > http://rubyforge.org/mailman/listinfo/redcloth-upwards-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/redcloth-upwards/attachments/20070823/5db4daa7/attachment-0001.html
At 4:37 PM -0400 8/23/07, Jason Garber wrote:>Unfortunately, I can''t come up with a list of the sets you request, Stephen. Off-hand, I know definition lists are done differently in RedCloth (not implemented in SRC) than in T2; backticks to denote code were in RC (don''t know about SRC).You just did come up with a list -- it''s partial and informal -- but that is a great place to start!>What I''d like to do is hook up another test that runs all the cases (yml files) through T2 (php) and compares the output. I don''t necessarily want that to be a permanent part of SRC''s test suite, but it would be interesting to see exactly how they handle things differently. I think I''ve encountered a few cases where T2 has undesirable output but SRC does it right. Obviously we''d want to preserve SRC''s way.I''d be interested in your tests and the results. Please post them to the list.
Okay, last night I hooked up the tests to Textile-2.0.0. Lots of failures, but it''s hard to talk about them in a meaningful way on this list (there''s a 40KB limit, so I can''t even attach the test failures). Rather than contriving examples and lots of copying and pasting into emails, I think I''d rather make changes to the tests and then everyone can look at the changeset on Trac or look through the YAML files if they want and make sure it''s agreeable to all. I will be quick to roll back some or all of my changes if there''s opposition, especially from _why. I''m not trying to do any code fisting here. :-) On Aug 23, 2007, at 4:49 PM, Stephen Bannasch wrote:> At 4:37 PM -0400 8/23/07, Jason Garber wrote: >> Unfortunately, I can''t come up with a list of the sets you >> request, Stephen. Off-hand, I know definition lists are done >> differently in RedCloth (not implemented in SRC) than in T2; >> backticks to denote code were in RC (don''t know about SRC). > > You just did come up with a list -- it''s partial and informal -- > but that is a great place to start! > >> What I''d like to do is hook up another test that runs all the >> cases (yml files) through T2 (php) and compares the output. I >> don''t necessarily want that to be a permanent part of SRC''s test >> suite, but it would be interesting to see exactly how they handle >> things differently. I think I''ve encountered a few cases where T2 >> has undesirable output but SRC does it right. Obviously we''d want >> to preserve SRC''s way. > > I''d be interested in your tests and the results. Please post them > to the list. > _______________________________________________ > Redcloth-upwards mailing list > Redcloth-upwards at rubyforge.org > http://rubyforge.org/mailman/listinfo/redcloth-upwards
I''m going to be working one YAML file at a time and posting my comments to the list. No substitute for reading the diffs and yml files yourself, but it helps to have a summary. I want to be very open about the changes. Changes: * Escaping needs to be different within code tags (or @ or bc.) * Removed triple backticks as a code signature * Switched tests that used <pre class="code> to bc. * Removed Markdown-style indented code * Added Textile2 double-equals block ignore * Removed the forced observance of inline signatures within code: here is some *starred* code but I want some redcloth*bold*redcloth code. * Leaving in HTML becoming <span class="caps">HTML</span>. I don''t see how it''s useful, but Textile2 does it. SRC doesn''t yet. * Added a test to ensure hard breaks aren''t inserted between HTML elements (like LIs or <pre>\n<code>) Remaining differences from Textile2: * Textile2 doesn''t seem to like multiple ==ignores== on one line * Textile2 uses numerical entities. I prefer SRC''s already-working named ones. * Textile2 doesn''t observe hard breaks inside HTML elements (see: "and it will not put a <br /> between them") * If something is wrapped in <notextile> tags, Textile2 still puts it in a paragraph You can see how actual SRC output measures up to expected output with rake test:parser:code and see how the expected output measures up to Textile 2.0.0''s output with rake test:textile2:code My AIM name is JasonGarberEMU if you want to chat.
Changes: none Differences: * Textile-2.0.0 (php) doesn''t support definition lists. MT-Textile (perl) does support a dl. term:definition format. RedCloth already has its own definition list format. Neither are implemented in SRC, but one ought to be. The question for you all is: which one?
SRC''s image handling works really well. All tests pass for the parser. Not many do against Textile2 because it includes an empty title attribute and it doesn''t handle image URLs near as well as SRC.
Changes: * Fixed missing hard breaks in expected output Questions: * A single dash - surrounded by spaces - comes out as &8211; but eats the spaces on either side. I know _why talked about this with em dashes and I''m so glad he had it eat spaces surrounding the em dash. Should en dashes also eat the surrounding spaces? * Take a look at how divs are opened (and closed?) in test_instiki_7 & 8. Textile2 nor MT-Textile support this and the way of closing seems unintuitive. I''d like to remove these tests and let div(foo). work just like p(foo). Anything more complex and you can just use HTML. Anyone disagree?
Hi Jason, Here''s a comment:>* Textile2 uses numerical entities. I prefer SRC''s already-working >named ones.Just which entities are you referring to? I''ve got a patch which fixes some problems I''ve been having and they both relate to entities in different situations. When an ampersand is used in a url and SRC translates it to the entity & this causes problems later for me with some html rendering engines. The following patch instead replaces the ampersand with the entity &. The second problem is that the I am also sending the xhtml output through some xml processing and enitites like the right arrow cause a problem unless I also declare the entity at the top of he xml document (which contains SRC output along with lost of other stuff). Instead I use the Unicode entity → for the arrow. Neither of these changes appear to cause any problems: Index: lib/superredcloth.rb ==================================================================--- lib/superredcloth.rb (revision 180) +++ lib/superredcloth.rb (working copy) @@ -82,7 +82,7 @@ def image opts p_opts = {:float => opts.delete(:align)} if opts[:align] opts[:alt] = opts[:title] - img = "<img src=\"#{opts[:src]}\"#{pba(opts)} alt=\"#{opts[:alt]}\" />" + img = "<img src=\"#{urlesc(opts[:src])}\"#{pba(opts)} alt=\"#{opts[:alt]}\" />" img = "<a href=\"#{urlesc opts[:href]}\">#{img}</a>" if opts[:href] img = "<p#{pba(p_opts)}>#{img}</p>" if p_opts img @@ -115,7 +115,7 @@ "–" end def arrow opts - "→" + ''→'' end def dim opts "#{opts[:x]}×#{opts[:y]}" @@ -133,6 +133,6 @@ "&#{opts[:text]};" end def urlesc txt - txt.gsub(/&/, ''&'') + txt.gsub(/&/, ''&'') end end
On Fri, Aug 24, 2007 at 11:21:17AM -0400, Jason Garber wrote:> I will be quick to roll back some or all of my changes if there''s > opposition, especially from _why. I''m not trying to do any code > fisting here. :-)Toosk toosk. On the contrary. You''re doing great and I''m verrry mentally occupied with Shoes and Hackety Hack. Just go with it, I''m really totally happy about it. _why
Stephen, I was referring to the translation of "Barnes & Noble" into "Barnes & Noble" in regular text. Textile2 does it as "Barnes & Noble" so I prefer the SuperRedCloth way of converting certain entities into the named entities instead of numerical ones (see superredcloth_inline.rl about line 209). It still doesn''t convert all entities though, so if I write a pound sign (?), it is not converted to £ or £. Should it be? I applied your patch. Thanks! On Aug 24, 2007, at 5:45 PM, Stephen Bannasch wrote:> Hi Jason, > > Here''s a comment: >> * Textile2 uses numerical entities. I prefer SRC''s already-working >> named ones. > > Just which entities are you referring to? > > I''ve got a patch which fixes some problems I''ve been having and > they both relate to entities in different situations. > > When an ampersand is used in a url and SRC translates it to the > entity & this causes problems later for me with some html > rendering engines. The following patch instead replaces the > ampersand with the entity &. > > The second problem is that the I am also sending the xhtml output > through some xml processing and enitites like the right arrow cause > a problem unless I also declare the entity at the top of he xml > document (which contains SRC output along with lost of other > stuff). Instead I use the Unicode entity → for the arrow. > > Neither of these changes appear to cause any problems: > > Index: lib/superredcloth.rb > ==================================================================> --- lib/superredcloth.rb (revision 180) > +++ lib/superredcloth.rb (working copy) > @@ -82,7 +82,7 @@ > def image opts > p_opts = {:float => opts.delete(:align)} if opts[:align] > opts[:alt] = opts[:title] > - img = "<img src=\"#{opts[:src]}\"#{pba(opts)} alt=\"#{opts > [:alt]}\" />" > + img = "<img src=\"#{urlesc(opts[:src])}\"#{pba(opts)} alt=\"# > {opts[:alt]}\" />" > img = "<a href=\"#{urlesc opts[:href]}\">#{img}</a>" if opts > [:href] > img = "<p#{pba(p_opts)}>#{img}</p>" if p_opts > img > @@ -115,7 +115,7 @@ > "–" > end > def arrow opts > - "→" > + ''→'' > end > def dim opts > "#{opts[:x]}×#{opts[:y]}" > @@ -133,6 +133,6 @@ > "&#{opts[:text]};" > end > def urlesc txt > - txt.gsub(/&/, ''&'') > + txt.gsub(/&/, ''&'') > end > end > _______________________________________________ > Redcloth-upwards mailing list > Redcloth-upwards at rubyforge.org > http://rubyforge.org/mailman/listinfo/redcloth-upwards
links.yml: nothing to change!
lists.yml:
Test 3 ("Funky...") doesn''t make a whole lot of sense to me.
I get
confused with all the nesting and type switching. Textile2 doesn''t
have a clue either. Someone else want to take a look at it and make
a recommendation?
Starting/continuation of numbering is the biggest thing broken.
test_lists_10 and 11 are failing because they try to continue
numbering within a list (not using the start attribute but handling
it with paragraph tags within the LI). I''m removing those parts
because I don''t think it''s something Super should do. Take a
look at
the diff and let me know if you disagree.
Jason Garber
2007-Aug-27 18:12 UTC
Test cleanup: poignant.yml & important alignment modifier question
Other than being crazy long, this test is okay. It raises a question I''ve had for awhile, though: what should alignment modifiers do? !>/image.gif! right-aligned image <p style="float: right"><img src="/image.gif" /> right-aligned image</ p> (Current SRC expected output) <p><p style="float: right;"><img src="/image.gif" /></p> right- aligned image</p> (Current SRC actual output) <p><img src="/image.gif" align="right" /> right-aligned image</p> (What Textile2 (Textism, WordPress, and MovableType-non-CSS do) <p><img src="/image.gif" class="right" /> right-aligned image</p> (What I would prefer) The align attribute is deprecated, but forcing a right float gives me the willies. What if you want to have some left margin on right- floating images and right margin on left-floating ones? You can''t really detect or override the float in your CSS. Is anyone opposed to changing the alignment modifiers to have effect on class and leaving it to the designer to write styles for those classes? Is there a better solution?
I cleaned up a bunch of duplicate rows in tables that were unnecessary for testing purposes. There''s one test that uses a period between vertical pipes to represent an empty table cell. The expected output shows |.| is rendered as <td> </td> but actual output leaves the dot. Textile2 leaves the dot too. If you remove the period from the input and just have two vertical pipes ||, Textile2 ignores the cell (and leaves the row one short) and SRC puts the pipe with the next cell, so that won''t work. The empty cell "feature" (|.|) is not documented anywhere and is unique to RedCloth. And it''s not implemented in SRC. Can it be dropped?
why the lucky stiff
2007-Aug-27 20:02 UTC
Test cleanup: poignant.yml & important alignment modifier question
On Mon, Aug 27, 2007 at 02:12:03PM -0400, Jason Garber wrote:> Is anyone opposed to changing the alignment modifiers to have effect > on class and leaving it to the designer to write styles for those > classes? Is there a better solution?What if we leave the default behavior to float? And then we allow an override to hook up to a CSS class, if they like. I the idea of using classes much better, but I''m worried about people not seeing instant results when they haven''t got the right CSS, you know? _why
Jason Garber
2007-Aug-27 21:25 UTC
Test cleanup: textism.yml & removing sanitize, filter, and hard_breaks tests
h2. textism.yml Broke the huge sample test from http://textism.com/tools/textile/? sample=2 into manageable chunks. Re-enabled the commented-out test of providing a named URL anywhere in the document and linking to it. Added footnote classes that are present in Textile2 but not in SRC. h2. Removing sanitize, filter, and hard_breaks tests I removed sanitize and filter tests since these are not options (yet?) in SuperRedCloth. Someone can reinstate them when they implement these features. I integrated the hard_breaks tests into textism.yml and removed it as a separate set of tests. This is the end of my cleaning up tests. Now to fix bugs! IM me if you''re going to be fixing some too. I''d love help! AIM: JasonGarberEMU And please give me your input on all the questions I asked in previous emails!
Jason Garber
2007-Aug-27 21:26 UTC
Test cleanup: poignant.yml & important alignment modifier question
Yes, that crossed my mind. Excellent suggestion! On Aug 27, 2007, at 4:02 PM, why the lucky stiff wrote:> On Mon, Aug 27, 2007 at 02:12:03PM -0400, Jason Garber wrote: >> Is anyone opposed to changing the alignment modifiers to have effect >> on class and leaving it to the designer to write styles for those >> classes? Is there a better solution? > > What if we leave the default behavior to float? And then we allow > an override to hook up to a CSS class, if they like. I the idea > of using classes much better, but I''m worried about people not > seeing instant results when they haven''t got the right CSS, you > know? > > _why > _______________________________________________ > Redcloth-upwards mailing list > Redcloth-upwards at rubyforge.org > http://rubyforge.org/mailman/listinfo/redcloth-upwards
On Mon, Aug 27, 2007 at 02:53:17PM -0400, Jason Garber wrote:> The empty cell "feature" (|.|) is not documented anywhere and is > unique to RedCloth. And it''s not implemented in SRC. Can it be dropped?I may have taken this from PyTextile, from which I pulled some of these tests. It''s so obscure and minor, I really don''t mind if it goes it away. _why
At 8:31 PM -0500 8/27/07, why the lucky stiff wrote:>On Mon, Aug 27, 2007 at 02:53:17PM -0400, Jason Garber wrote: > > The empty cell "feature" (|.|) is not documented anywhere and is >> unique to RedCloth. And it''s not implemented in SRC. Can it be dropped? > >I may have taken this from PyTextile, from which I pulled some of >these tests. It''s so obscure and minor, I really don''t mind if it >goes it away.I can''t think of a reason to need an empty cell: ''|.|'' <td></td> that can''t be filled by a cell with one space: ''| |'' => <td> </td>
At 11:55 AM -0400 8/27/07, Jason Garber wrote:>Stephen, I was referring to the translation of "Barnes & Noble" into >"Barnes & Noble" in regular text. Textile2 does it as "Barnes >& Noble" so I prefer the SuperRedCloth way of converting certain >entities into the named entities instead of numerical ones (see >superredcloth_inline.rl about line 209).I agree, however I''m wrestling with whether SRC should only use named entities for the first four of the five standard XML character entity references http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references#Character_entities_in_XML & & (ampersand, U+0026) < < (left angle bracket, less-than sign, U+003C) > > (right angle bracket, greater-than sign, U+003E) " " (quotation mark, U+0022) ' '' (apostrophe, U+0027) These are the only named entities that xml documents can contain unless the character entities are declared. The apostrophe entity isn''t actually a standard html entity so only the first four should be used. If the output of SRC is going to be processed by any other general XML processor any other special character references should be converted to numerical Unicode character references. We are using an XML processing path that chokes on entity references like →. However we are specifically including XHTML content in our XML documents and right now we are not including the XHTML DTD character entity declarations -- and now I''m thinking we should include these declarations. See: http://www.w3.org/TR/xhtml1/dtds.html#h-A2 Which references these three DTD entity files: http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent You applied my patch which removes → (which fixes my current problem). The only other character entity in SRC is in test/table.yml in the output around line 180. The nbsp is the output for ''|.|''. However I don''t even see any code for producing the nbsp enity anymore so you could probably replace the test with ''| |'' and look for <td> </td>.>It still doesn''t convert >all entities though, so if I write a pound sign (?), it is not >converted to £ or £. Should it be?I''m not sure what is best. When should a ''special'' character like ? be converted to an entity? It''s not part of the document structure like ''<'' so we don''t need to convert it for that reason. I don''t think it''s valid in a url so it shouldn''t appear there at all. In a document: If you convert it to a character entity it will be human parsable but it will cause a generic XML processor to choke. If you convert it to a numerical character referenceI think it will work everywhere but it won''t be easily parsable. If you leave it un-translated AND transport protocols don''t mess with it AND it''s displayed properly to users then it''s probably best not to translate it at all.