I'm currently attempting to write a spec for parsing Markdown Extra, and since one goal is to minimize the differences in output between implementations, I've made a tool allowing me to compare who does what for any given input. I hope this can also facilitate future discussions about the syntax. So here's Babelmark, a testbed for various Markdown implementations were you give the input and you get the HTML output for Markdown.pl 1.0.1 and 1.0.2b8, PHP Markdown and PHP Markdown Extra, Python Markdown, Text::Markdown and Text::MultiMarkdown, and finally Showdown. Unfortunatly, my web host doesn't do Ruby, nor Java, C# or Lua, so the online version is missing a couple of interesting implementations. Locally on my computer Babelmark also do BlueCloth, Maruku, MarkdownJ, markdown.lua, and Pandoc. I'm very sorry if your Markdown implementation can't be part of Babelmark online, but if anyone has a better host to offer for Babelmark, ideally with support for all of these, I'd gladly send him the scripts. You can find Babelmark here: <http://michelf.com/projects/babelmark/> Also note that, contrary to the normal Markdown or PHP Markdown dingus, you can bookmark specific tests since the input string is encoded in the URI. Here are some interesting cases for instance: <http://michelf.com/projects/babelmark/?markdown=_test_test_test_> <http://michelf.com/projects/babelmark/?markdown=*test+%5Btest*+test%5D%28%23%29 > <http://michelf.com/projects/babelmark/?markdown=%5Btest%5D%0D%0A%0D%0A%5Btest... > So that's it. Have fun with it. Michel Fortin michel.fortin at michelf.com http://michelf.com/
This is a great tool. It won't solve all of our problems by itself (we still need a spec), but I think it will help a great deal if we want to try to bring all the implementations into agreement. Some of my tests that show interesting differences: <%foo ok %>: http://michelf.com/projects/babelmark/?markdown=%3C%25foo+ok+%25%3E (BTW, where does one get Markdown.pl 1.0.2 beta? Markdown website says that the last version is 1.0.1, of 2004.) foo_bar_bas: http://michelf.com/projects/babelmark/?markdown=foo_bar_bas 2-tier list indented by three spaces: http://michelf.com/projects/babelmark/?markdown=*+what%27s+up%3F%0D%0A+++*+ok Now, on this one, I must say I have mixed feelings, since python-markdown is the only implementation that follows "Markdown Syntax" and treats the item indented by three spaces as being at the same level. Makes me feel like a naive fool for following the "spec". :) One more thing: python-markdown is a stickler for explicit encoding (because we rely on it for bidi-detection). The input must be either python Unicode object or UTF8 encoded byte-string or you must say what encoding you are using. Your test doesn't seems to send something other than UTF8 without specifying encoding. http://michelf.com/projects/babelmark/?markdown=_%D8%A7%D9%84%D9%92%D8%B9%D9%...+ Should we start a page on the wiki to collect those examples so that we can then discuss what ought to be done in each case? - yuri On Sat, Mar 22, 2008 at 10:09 AM, Michel Fortin <michel.fortin at michelf.com> wrote:> I'm currently attempting to write a spec for parsing Markdown Extra, > and since one goal is to minimize the differences in output between > implementations, I've made a tool allowing me to compare who does what > for any given input. I hope this can also facilitate future > discussions about the syntax.-- http://sputnik.freewisdom.org/
* Yuri Takhteyev <qaramazov at gmail.com> [2008-03-22 19:00]:> (BTW, where does one get Markdown.pl 1.0.2 beta? Markdown > website says that the last version is 1.0.1, of 2004.)http://daringfireball.net/projects/downloads/Markdown_1.0.2b8.tbz Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Le 2008-03-22 ? 13:57, Yuri Takhteyev a ?crit :> 2-tier list indented by three spaces: > http://michelf.com/projects/babelmark/?markdown=*+what%27s+up%3F%0D%0A+++*+ok > > Now, on this one, I must say I have mixed feelings, since > python-markdown is the only implementation that follows "Markdown > Syntax" and treats the item indented by three spaces as being at the > same level. Makes me feel like a naive fool for following the "spec". > :)Well, you've been following the official spec; no one should call you a fool for that. But it certainly doesn't give much leverage to the idea of keeping the spec as it is.> One more thing: python-markdown is a stickler for explicit encoding > (because we rely on it for bidi-detection). The input must be either > python Unicode object or UTF8 encoded byte-string or you must say what > encoding you are using. Your test doesn't seems to send something > other than UTF8 without specifying encoding. > > http://michelf.com/projects/babelmark/?markdown=_%D8%A7%D9%84%D9%92%D8%B9%D9%...+Ah, good catch. Should be fixed now. Note that I haven't been using Python Markdown's script directly since I didn't find how to use the standard input with it. So I've written a 4-line python stub which does the job of collecting the input and calling the markdown function inside markdown.py. Here's the script (after the fix): import sys, codecs, markdown text = unicode(sys.stdin.read(), "utf-8") html = markdown.markdown(text) sys.stdout.write(html.encode("utf-8"))> Should we start a page on the wiki to collect those examples so that > we can then discuss what ought to be done in each case?Sure, we could... But I think we already have half of this in the form of a testsuite, and the other half in the form of this email discussion list. As I said already, I'm open to adding new tests to MDTest, things like the above for instance, and I'm open to discussion about what the expected output should be. If you want, you could build a separate testsuite for Python Markdown that I'll add to the next release of MDTest. The more implementations can pass each other's testsuite the better interoperability we'll have achieved. Michel Fortin michel.fortin at michelf.com http://michelf.com/
* Michel Fortin <michel.fortin at michelf.com> [2008-03-22 22:30]:> The more implementations can pass each other's testsuite the > better interoperability we'll have achieved.(Assuming, of course, that the test suite covers enough edge and corner cases.) Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
On 22 Mar 2008, at 21:47, Aristotle Pagaltzis wrote:> * Michel Fortin <michel.fortin at michelf.com> [2008-03-22 22:30]: >> The more implementations can pass each other's testsuite the >> better interoperability we'll have achieved. > > (Assuming, of course, that the test suite covers enough edge and > corner cases.)I'd say that that is directly proportional to the utility of the test suite ;-) I've got [Devel::Cover] to measure coverage, and it (on [my trunk], which is close to 1.0.17) says that my coverage is almost perfect: File stmt bran cond sub time total ---------------------------- ------ ------ ------ ------ ------ ------ blib/lib/Text/Markdown.pm 99.8 94.2 72.0 100.0 100.0 87.0 96.9 Which is *just wrong* - as I know there are piles of corner cases that my tests don't adequately cover. :( Probably a lesson to point out that automated coverage analysis isn't too useful in some cases... [Devel::Cover]: http://search.cpan.org/~pjcj/Devel-Cover-0.63/lib/ Devel/Cover.pm [my trunk]: http://svn.kulp.ch/cpan/text_multimarkdown/trunk/ Cheers Tom
* Tomas Doran <bobtfish at bobtfish.net> [2008-03-23 13:10]:> I've got [Devel::Cover] to measure coverage, and it (on [my > trunk], which is close to 1.0.17) says that my coverage is > almost perfect: > > File stmt bran cond sub time total > ---------------------------- ------ ------ ------ ------ ------ ------ > blib/lib/Text/Markdown.pm 99.8 94.2 72.0 100.0 100.0 87.0 > 96.9 > > Which is *just wrong* - as I know there are piles of corner > cases that my tests don't adequately cover. :( > > Probably a lesson to point out that automated coverage analysis > isn't too useful in some cases... > > [Devel::Cover]: > http://search.cpan.org/~pjcj/Devel-Cover-0.63/lib/Devel/Cover.pm > [my trunk]: http://svn.kulp.ch/cpan/text_multimarkdown/trunk/Yes, well. If you write one test case that causes the code to perform a particular pattern match, then the match expression is considered covered? even if uses a 500-line regex that parses an entire file format. This problem basically occurs whenever a separate language is embedded into the main code: SQL queries, XPath expressions, templates in some template mini-language, whatever. If the coverage metric doesn?t cross the language barrier right along with the code, it breaks down at these points. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Le 2008-03-22 ? 17:27, Michel Fortin a ?crit :>> 2-tier list indented by three spaces: >> http://michelf.com/projects/babelmark/?markdown=*+what%27s+up%3F%0D%0A+++*+ok >> >> Now, on this one, I must say I have mixed feelings, since >> python-markdown is the only implementation that follows "Markdown >> Syntax" and treats the item indented by three spaces as being at the >> same level. Makes me feel like a naive fool for following the >> "spec". >> :) > > Well, you've been following the official spec; no one should call > you a fool for that. But it certainly doesn't give much leverage to > the idea of keeping the spec as it is.It should be mentioned that, in addition to Python Markdown, both markdown.lua and Pandoc seem to follow the spec regarding list indentation. Michel Fortin michel.fortin at michelf.com http://michelf.com/
On 24 Mar 2008, at 03:11, Michel Fortin wrote:> Le 2008-03-22 ? 17:27, Michel Fortin a ?crit : > >>> 2-tier list indented by three spaces: >>> http://michelf.com/projects/babelmark/?markdown=*+what%27s+up%3F% >>> 0D%0A+++*+ok >>> >>> Now, on this one, I must say I have mixed feelings, since >>> python-markdown is the only implementation that follows "Markdown >>> Syntax" and treats the item indented by three spaces as being at >>> the >>> same level. Makes me feel like a naive fool for following the >>> "spec". >>> :) >> >> Well, you've been following the official spec; no one should call >> you a fool for that. But it certainly doesn't give much leverage >> to the idea of keeping the spec as it is. > > It should be mentioned that, in addition to Python Markdown, both > markdown.lua and Pandoc seem to follow the spec regarding list > indentation.I've transplanted Babelmark, and got all the implementations running online. I've also added support for Discount (David Parson's C markdown implementation announced recently). http://babelmark.bobtfish.net/?markdown=*+what%27s+up%3F%0D%0A+++* +ok&normalize=on Cheers Tom
Terrific, that's really useful! For comparison purposes, you may want to run pandoc with the --strict flag, to turn off syntax extensions (or use the compatibility script hsmarkdown, included with the pandoc distribution). Or, if you want, include both strict and extended versions. Best, John> I've transplanted Babelmark, and got all the implementations running > online. > > I've also added support for Discount (David Parson's C markdown > implementation announced recently). > > http://babelmark.bobtfish.net/?markdown=*+what%27s+up%3F%0D%0A+++*+ok&nor... > > Cheers > Tom > > _______________________________________________ > Markdown-Discuss mailing list > Markdown-Discuss at six.pairlist.net > http://six.pairlist.net/mailman/listinfo/markdown-discuss >
On 24 Mar 2008, at 15:34, John MacFarlane wrote:> Or, if you want, > include both strict and extended versions.Done: http://babelmark.bobtfish.net/?markdown=Test+footnote.%5B%5E1%5D%5B% 5D.%0D%0A%0D%0A%5B%5E1%5D%3A+This+is+a+footnote.%0D%0A&normalize=on Cheers Tom
Here are some of my personal favorites, with lots of divergence between implementations: http://babelmark.bobtfish.net/?markdown=-+foo%0D%0A-+bar%0D%0A%0D%0A1.+first%... http://babelmark.bobtfish.net/?markdown=%2B+++item+1%0D%0A%0D%0A++++%2B+++ite... http://babelmark.bobtfish.net/?markdown=+8.+item+1%0D%0A+9.+item+2%0D%0A10.+i... http://babelmark.bobtfish.net/?markdown=x%3Cmax(a%2Cb)%0D%0A&normalize=on John
On Mar 24, 2008, at 4:47 PM, John MacFarlane wrote:> Here are some of my personal favorites, with lots of divergence > between implementations:Here is another funny one: http://babelmark.bobtfish.net/?markdown=_M*A*S*H_&normalize=on Great resource :) -- PA. http://alt.textdrive.com/nanoki/
Any reason for including Text::MultiMarkdown and not the official MultiMarkdown itself? It would be useful for finding any areas where the two versions differ from each other, in addition to where MMD differs from other implementations. This looks like a fantastic tool, in either case - thanks for sharing it. F- On Mar 22, 2008, at 1:09 PM, Michel Fortin wrote:> I'm currently attempting to write a spec for parsing Markdown Extra, > and since one goal is to minimize the differences in output between > implementations, I've made a tool allowing me to compare who does > what for any given input. I hope this can also facilitate future > discussions about the syntax. > > So here's Babelmark, a testbed for various Markdown implementations > were you give the input and you get the HTML output for Markdown.pl > 1.0.1 and 1.0.2b8, PHP Markdown and PHP Markdown Extra, Python > Markdown, Text::Markdown and Text::MultiMarkdown, and finally > Showdown. > > Unfortunatly, my web host doesn't do Ruby, nor Java, C# or Lua, so > the online version is missing a couple of interesting > implementations. Locally on my computer Babelmark also do BlueCloth, > Maruku, MarkdownJ, markdown.lua, and Pandoc. I'm very sorry if your > Markdown implementation can't be part of Babelmark online, but if > anyone has a better host to offer for Babelmark, ideally with > support for all of these, I'd gladly send him the scripts. > > You can find Babelmark here: > > <http://michelf.com/projects/babelmark/> > > Also note that, contrary to the normal Markdown or PHP Markdown > dingus, you can bookmark specific tests since the input string is > encoded in the URI. Here are some interesting cases for instance: > > <http://michelf.com/projects/babelmark/?markdown=_test_test_test_> > <http://michelf.com/projects/babelmark/?markdown=*test+%5Btest*+test%5D%28%23%29 > > > <http://michelf.com/projects/babelmark/?markdown=%5Btest%5D%0D%0A%0D%0A%5Btest... > > > > So that's it. Have fun with it. > > > Michel Fortin > michel.fortin at michelf.com > http://michelf.com/ > > > _______________________________________________ > Markdown-Discuss mailing list > Markdown-Discuss at six.pairlist.net > http://six.pairlist.net/mailman/listinfo/markdown-discuss-- Fletcher T. Penney fletcher at fletcherpenney.net I had just received my degree in Calcium Anthropology...the study of milkmen. - Steven Wright -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2437 bytes Desc: not available Url : <http://six.pairlist.net/pipermail/markdown-discuss/attachments/20080322/b3514...>
Le 2008-03-22 ? 21:15, Fletcher T. Penney a ?crit :> Any reason for including Text::MultiMarkdown and not the official > MultiMarkdown itself?Hum, because I just took what was bundled with Text::Markdown; it was easier that way and I though it was the same thing just packaged differently. Perhaps I shouldn't have.> It would be useful for finding any areas where the two versions > differ from each other, in addition to where MMD differs from other > implementations.But should we expect any difference? I agree that I should be using the official MultiMarkdown -- and I've added it now --, but is it still worth keeping Text::MultiMarkdown then?> This looks like a fantastic tool, in either case - thanks for > sharing it.You're welcome, and I'm glad you like it. Michel Fortin michel.fortin at michelf.com http://michelf.com/
On 23 Mar 2008, at 03:50, Michel Fortin wrote:> Le 2008-03-22 ? 21:15, Fletcher T. Penney a ?crit : > >> Any reason for including Text::MultiMarkdown and not the official >> MultiMarkdown itself? > > Hum, because I just took what was bundled with Text::Markdown; it > was easier that way and I though it was the same thing just > packaged differently. Perhaps I shouldn't have.It's the same thing in the same way that Text::Markdown is "just markdown, packaged differently".> >> It would be useful for finding any areas where the two versions >> differ from each other, in addition to where MMD differs from >> other implementations. > > But should we expect any difference? I agree that I should be using > the official MultiMarkdown -- and I've added it now --, but is it > still worth keeping Text::MultiMarkdown then? >Yes, please also keep Text::MultiMarkdown. This is actually *much more* different to fletcher's MultiMarkdown than Text::Markdown is to Markdown.pl - as with my refactoring, MultiMarkdown is able to just overload / hook into Text::Markdown (as I've created the relevant hook points), as opposed to being a copy & paste of original markdown with bits stuck on the side.. (No offense meant here - I'm only able to do this as I'm *also* maintaining a fork of Markdown). Again, I'm aiming form *zero difference* between Text::MultiMarkdown and 'official' MultiMarkdown, except for bugs that I've fixed (so we shouldn't *expect* difference, and if it's found, and I don't have an explicit test in my distribution to cover it - it's a bug in my code and I *will fix it*).. My code does pass all of Fletcher's test suite, so I'm sure it's pretty similar. ;) As previously noted, s/MultiMarkdown/Markdown/g; s/Fletcher/John/g; in the paragraph above is also be true. Cheers Tom
Agreed. Assuming that it's minimal effort to leave both in, I think it's better to leave as many options in babelmark as possible. When I add a new feature or "fix" something in MultiMarkdown, it may break something else, or it may work differently in my implementation than in Tomas'. By having these different flavors in one place, it will make it much easier to keep small unintentional "drifts" from occurring between "dialects". Ideally, the only differences would occur as related to syntax features that are implemented in one dialect, and not in another. Another benefit is the ability to merge similar features that are implemented differently. For example, when Michel added footnote support to PHP Markdown Extra that was almost the same as the approach I had used in MultiMarkdown, it was easy for me to change the XHTML syntax so that they were compatible (though it appears they have drifted apart again...) F- On Mar 23, 2008, at 8:13 AM, Tomas Doran wrote:> > Yes, please also keep Text::MultiMarkdown. This is actually *much > more* different to fletcher's MultiMarkdown than Text::Markdown is > to Markdown.pl - as with my refactoring, MultiMarkdown is able to > just overload / hook into Text::Markdown (as I've created the > relevant hook points), as opposed to being a copy & paste of > original markdown with bits stuck on the side.. (No offense meant > here - I'm only able to do this as I'm *also* maintaining a fork of > Markdown). > > Again, I'm aiming form *zero difference* between Text::MultiMarkdown > and 'official' MultiMarkdown, except for bugs that I've fixed (so we > shouldn't *expect* difference, and if it's found, and I don't have > an explicit test in my distribution to cover it - it's a bug in my > code and I *will fix it*).. My code does pass all of Fletcher's test > suite, so I'm sure it's pretty similar. ;) > > As previously noted, s/MultiMarkdown/Markdown/g; s/Fletcher/John/g; > in the paragraph above is also be true. > > Cheers > Tom-- Fletcher T. Penney fletcher at fletcherpenney.net On two occasions I have been asked [by members of Parliament], 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question. - Charles Babbage -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2437 bytes Desc: not available Url : <http://six.pairlist.net/pipermail/markdown-discuss/attachments/20080323/32745...>
One problem with babelmark - the CSS is such that a long line of output (or input) forces the body off the left side of the page. For an example, try the following text in the babelmark text source: Test footnote.[^1][]. [^1]: This is a footnote. F- -- Fletcher T. Penney fletcher at fletcherpenney.net Failure is the condiment that gives success its flavor. - Truman Capote -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2437 bytes Desc: not available Url : <http://six.pairlist.net/pipermail/markdown-discuss/attachments/20080323/43df4...>
On Sun, Mar 23, 2008 at 5:44 AM, Fletcher T. Penney <fletcher at fletcherpenney.net> wrote:> One problem with babelmark - the CSS is such that a long line of > output (or input) forces the body off the left side of the page. > > > For an example, try the following text in the babelmark text source: > > Test footnote.[^1][]. > > [^1]: This is a footnote.or this: http://michelf.com/projects/babelmark/?markdown=aaaaaaaaaaaaaaaaaaaaaaaaaaaaa...
Le 2008-03-23 ? 8:44, Fletcher T. Penney a ?crit :> One problem with babelmark - the CSS is such that a long line of > output (or input) forces the body off the left side of the page.You're right. In fact, this problem also applies to the PHP Markdown Dingus since it uses the same stylesheet. I'll see if I can improve the layout. Thank you for mentioning this. Michel Fortin michel.fortin at michelf.com http://michelf.com/
On 22 Mar 2008, at 17:09, Michel Fortin wrote:> I'm currently attempting to write a spec for parsing Markdown > Extra, and since one goal is to minimize the differences in output > between implementations, I've made a tool allowing me to compare > who does what for any given input. I hope this can also facilitate > future discussions about the syntax. > > So here's Babelmark, a testbed for various Markdown implementations > were you give the input and you get the HTML output for Markdown.pl > 1.0.1 and 1.0.2b8, PHP Markdown and PHP Markdown Extra, Python > Markdown, Text::Markdown and Text::MultiMarkdown, and finally > Showdown. >This is really totally awesome :)> Unfortunatly, my web host doesn't do Ruby, nor Java, C# or Lua, so > the online version is missing a couple of interesting > implementations. Locally on my computer Babelmark also do > BlueCloth, Maruku, MarkdownJ, markdown.lua, and Pandoc. I'm very > sorry if your Markdown implementation can't be part of Babelmark > online, but if anyone has a better host to offer for Babelmark, > ideally with support for all of these, I'd gladly send him the > scripts. >I should be able to do all (or at least most of) the above, can I get a copy of the code (or access to the source control where it's located?). It would be interesting to be able to track 'should be like' implementations (for me, especially) - what would be good is some system that emailed me 'aggh' emails every time someone typed something and Text::Markdown was different to Markdown.pl.. However that would be trivially easy to spam me with, so might need some careful thought. ;) Cheers Tom
> Unfortunatly, my web host doesn't do Ruby, nor Java, C# or Lua, so the > online version is missing a couple of interesting implementations.If your host allows you to ssh and has make, then building Lua is very simple: mkdir ~/lua # or some other directory wget http://sputnik.freewisdom.org/files/easy-lua-install-2007-01-18 bash easy-lua-install-2007-01-18 --without-readline --prefix=/home/yuri/lua/ cd share/lua/5.1/ wget www.frykholm.se/files/markdown.lua ~/lua/bin/lua This will get you the interactive prompt: Lua 5.1.2 Copyright (C) 1994-2007 Lua.org, PUC-Rio> require"markdown" > =markdown("oh _hai_")<p>oh <em>hai</em></p>> Locally on my computer Babelmark also do BlueCloth, Maruku, MarkdownJ, > markdown.lua, and Pandoc. I'm very sorry if your Markdown > implementation can't be part of Babelmark online, but if anyone has a > better host to offer for Babelmark, ideally with support for all of > these, I'd gladly send him the scripts.I can try and see how far I get on webfaction. Can you send me the scripts? - yuri -- http://sputnik.freewisdom.org/