Tools

Lennart Kats, Eelco Visser and myself just got a paper accepted to LDTA'09. The paper is about declarative languages for describing programming editors. The main part of it is Lennart's work,  but it's running on top of the Spoofax transformation infrastructure. The idea is simple: You don't want to fight with Java, complicated APIs and complicated XML when you implement an Eclipse-based editor for your DSL. Instead, you describe your language's grammar with SDF, provide some auxiliary information using our declarative editor languages, and Spoofax/IMP does the rest by generating the editor engine for you.

The abstract explains it in the usual academic style:

Modern IDEs increase developer productivity by incorporating many different kinds of editor services. These can be purely syntactic, such as syntax highlighting, code folding, and an outline for navigation; or they can be based on the language semantics, such as in-line type error reporting and resolving identifier declarations. Building all these services from scratch requires both the extensive knowledge of the sometimes complicated and highly interdependent APIs and extension mechanisms of an IDE framework, and an in-depth understanding of the structure and semantics of the targeted language.

We discuss two examples: an editor plugin for WebDSL, a domain-specific language for web applications, and the embedding of WebDSL in Stratego, used for expressing the semantic rules of WebDSL.

Once I get bibtex-tools running on 64bit again, I'll link to the bib and pdf.

This last week, I spent some free cycles hacking together a small project instantiation tool for Stratego/XT. It makes setting up a fresh Stratego project really simple by automatically populating the project space with a default directory layout, build system files and some minimal program and syntax samples.

To create a project p0, all you have to do is:

$ crap --new-project p0

This creates all the files necessary for a complete GNU Autotools-based build system, including a sample Stratego program (src/xmpl.str):

p0/
   Makefile.am
   README.Developer
   README
   AUTHORS
   bootstrap
   p0.spec.in
   NEWS
   p0.pc.in
   configure.ac
   ChangeLog
   xmpl/
        Makefile.am
   syn/
       Makefile.am
   tests/
         Makefile.am
   src/
       Makefile.am
       xmpl.str

Once this is done, you can configure and compile the project,

$ ./bootstrap
$ ./configure
$ make all

install it,

$ make install

and even run the example transformation program:

$ echo "foo" | /usr/local/bin/xmpl
"Hello, World!"

The example program expects an input on stdin [or in a file specified by the -i switch], and will always produce the output string "Hello, World!".

The crap tool is part of the strategoxt-utils package. You can also download a stand-alone snapshot of crap.

More comprehensive documentation is available in the wiki. The tool is still very rough, so any suggestions for improvements and bug reports are very welcome.

I take strolls with my roommate, Håvard, quite often. Today, we touched on the subject of ODF vs OOXML after a longer discussion of document validation in general. That's when we hit upon the idea of the OOXML-validating mailfilter.

The idea is quite simple: In addition to your anti-virus, spamfilter, greylisting, and other filtering functionality you have on your mailserver, you add a small script which relies on unzip and xsltproc to validate any OOXML document that passes your way, and automatically rejects any non-conforming document with a stern warning, such as:

It appears that the document <file name>, created with
<document producing program>, is corrupted. I will not be able to
read this document properly. Please consider resending a conforming
OOXML document, or use a different document format, such as ODF.

Pedantically yours,

<your name>

I finally figured out how to add proper navigation history support to Spoofax today. This one has been bugging me for quite some time. I remember spending far too much time diving through the documentation with the hopes of figuring out how this should be done properly. No luck.

Today I had a flash of inspiration, so I dug into the JDT code base. That code seemed to solve the same problem in a very complicated way, so I didn't want to copy their approach outright. Stymied, I started tracing exactly what happens with the navigation history when positions are placed into it. After a bit of fiddling around, I figured out that when I move the cursor, I should mark the position both before and after the cursor/focus moves to get the behaviour of JDT (which I tried to emulate). I've always only tried saving the editor location state either before I changed it, or afterwards. I also tried all kinds of alternative calls on the EditorPart hierarchy in vain. I now use ITextEditor.setHighlightRange() which appears to do the job, provided I call markInNavigatorHistory() "properly".

Anyway, the lesson is simple: if you call AbstractTextEditor.markInNavigationHistory(), remember to do it twice -- once before you change the editor/focus and once afterwards.

It's official: I'm the bootstrapper. My hacking life in the last few weeks have hardly been anything but bootsrapping. I've already said a few things about the Stratego compiler hacking. Since it takes ~3-4 hours for a full build of the Stratego compiler in the Delft buildfarm, I've had a couple of other projects to dive into in parallell. One of these has been the porting of Eclipse IMP from Eclipse 3.2 to 3.3.

In short, IMP is an IDE generator based on Eclipse. It provides set of plugins and wizards that makes the development of programming language environments (a lot) easier. The basic workflow when building an IDE for you favourite language with IMP is, (1) provide a grammar defined using the LPG grammar language, (2) use the IMP-provided wizards inside Eclipse to generate things like syntax highlighting support, outline support, code folding support, templates, text hovers, etc, then (3) fill in the skeletons provided by the generator. My personal view (subject to change without warning) of the generated code is that it's a guide to which parts of the Eclipse framework you need to extend in order to provide a given piece of functionality. Sort of a little helpful gnome pointing you in the right direction. In some cases, the generated code will actually do all you want, but more often than not, you will want to go beyond it.

That was the backgrounder on IMP. A major drawback of the current IMP releases is that they will only work on 3.2. Oh, and, of course, that IMP requires IMP to build IMP. Getting this beast ported to 3.3 wasn't as straightforward as I'd hoped. It took a few iterations. The first was getting it to build properly without any problems on my plain 3.2 installation. That took me several days. All kinds of subtle bugs surfaced, presumably because I have a different set of development habits than the IMPers.

Once those were patched and fixed upstream, I managed to bootstrap my first version on 3.2. An ensuing battle with race conditions in the startup code of various plug-ins followed. I hate static initializers, but apparently not everybody does. In a multi-plugin architecture where the order to plugin loading is not guaranteed, I cannot see how you can safely assume the order of static initializers across plugins, but those questions are not for me to ponder. I ripped them out, and replaced them with lazy initializers as far as possible, and that worked wonders. With that hurdle out of the way, it was all down hill: a couple of internal JFace and JDT classes had changed locations and APIs between 3.2 and 3.3, but it was quick enough to rewrite the offending code (another reason why depending on internal APIs is a bitch, though I realize that the features in question could not have been provided without doing so).

It's a huge disappointment to realize that my patches are only a couple of hundreds of lines. I felt like I had to rewrite the world, at places... Anyway, here's hoping to its inclusion in one of the pending releases. I've updated our sdf2imp tool to use the 3.3-based IMP, so we're already seeing a return on my investment:)

I decided to forgo my friday night clubbing and rather spend the time at the office in solitude with a bottle of water, some microwave food and Eclipse windows floating feverishly about on my dualhead setup (I don't have an extra monitor at home).

The end result was another release of the Spoofax editor, codenamed Obelix. The name is apt; Obelix is 322% larger than Asterix (0.2.0) and weighs in at around 87KB. In addition to more crud, more features were also added:

• An outline view for Stratego files

  At the present time, the outliner will only update itself whenever you save. I have not added an autotimeout for reparsing, but this shouldn't be all that difficult.

  By clicking on the items in the outline view, the focus of the connected editor will be changed automatically, and the area in the source file associated with the declaration will be marked in the left hand side margin of the editor. Unfortunately, due to parsing inaccuracies, only the area corresponding to the declaration of the identifier will be marked, not all of its body too. I hope we can fix this soon.

  The information from provided by the outline is kept in the StrategoModel class, which I hope will evolve into a proper AST as time goes by and we improve our parser.
  

• Content assistance for strategies, rules and constructors
  

  

  Whenever you press Ctrl-Space, the content assistant will be run. It will consider the location you are in the document, parse leftwards and see if it can find something resembling strategy, rule or constructor identifier.

  When the scanner finds an identifier, it will assume it's a rule or strategy, and will show you all potential expansions for the prefix it found. By selecting one, it will insert the selection and move the cursor to after the completion. If you complete a higher-order strategy or rule, it will add a set of parentheses and place the cursor inside these.

  If, while scanning leftwards, it finds a !, it will assume you want a list of constructors.

  By typing ! or <, then waiting 500msec, the editor will automatically pop up a list of constructors or rule/strategy-definitions, respectively, all by itself.

• Preference pages for syntax highlighting
  

  

  Now the editor does not have to be restarted after changing the settings anymore. They are applied instantly.

• A nifty realtime parenthesis matcher
  

The parenthesis matcher keeps track of the kinds of parenthesis used, and if they don't match, it will signal a mismatch by a red (customizable) match cursor.

The interesting part of this -- content assist and outliner -- was made possible thanks to the new "outline" parser which is somewhat context sensitive. It skimps along the document trying to pick out anything it can reasonably detect, and places it into the StrategoModel. There's a long way left before we have 100% correct parsing. In particular, the unrestricted syntax embedding allowed in Stratego source code is of course not generally solvable. But as we approach there, we will have increasingly better completion and navigation.

A few words of acknowledgement are definitely in order. Inspiration by Martin helped a lot. On the technical side, looking at the SchemeWay project by Dominique Boucher was extremely helpful. My parenthesis matcher is heavily inspired by his. Also, I can heartily recommend the JavaEditor example in the Examples SDK to nail the basics of Eclipse editor development.

I released an update to the Stratego/XT editor plugin part of Spoofax today. This new version adds simple syntax highlighting support for SDF files.

I am still in the process of rewriting the Stratego parser to enable code completion and outlines. Further, I've started thinking about how to couple the editor with the compiler to map back Stratego compiler errors. I think this would easy as π if strc were to emit the errors as an ATerm.

All of this is slow going, as I have more pressing matters to finish.

To test the plugin provide the URL
http://www.ii.uib.no/~karltk/spoofax/
to the Eclipse Update Manager.

An enticing screenshot:

In an attempt at returning to normalcy after over a week of nothing but frantic documentation writing (in the form of an article), I have spent this week looking at module systems, modularization techniques and known methods for exposing variation points in these.

Apart from reading papers, I decided to experiment with extending frameworks, and I also had another itch to scratch. The result is a very rudimentary syntax highlighting text editor plugin for Eclipse, for the Stratego language.

For now, I've put it up on the temporary front page of spoofax.org, and I even provided a screenshot.

If you run Eclipse 3.0 or newer, you can download and install it using the Update Manager, as usual.

A lot of things are still missing, the most important (to me) of which are: context assist for strategies, rules, variables and constructors; outlining of strategy and rule definitions; a sensible indentation strategy; and semantic highlighting of variables in rule conditions and strategy expressions.