I finally figured out how to add proper navigation history support to Spoofax today. This one has been bugging me for quite some time. I remember spending far too much time diving through the documentation with the hopes of figuring out how this should be done properly. No luck.

Today I had a flash of inspiration, so I dug into the JDT code base. That code seemed to solve the same problem in a very complicated way, so I didn't want to copy their approach outright. Stymied, I started tracing exactly what happens with the navigation history when positions are placed into it. After a bit of fiddling around, I figured out that when I move the cursor, I should mark the position both before and after the cursor/focus moves to get the behaviour of JDT (which I tried to emulate). I've always only tried saving the editor location state either before I changed it, or afterwards. I also tried all kinds of alternative calls on the EditorPart hierarchy in vain. I now use ITextEditor.setHighlightRange() which appears to do the job, provided I call markInNavigatorHistory() "properly".

Anyway, the lesson is simple: if you call AbstractTextEditor.markInNavigationHistory(), remember to do it twice -- once before you change the editor/focus and once afterwards.

It's official: I'm the bootstrapper. My hacking life in the last few weeks have hardly been anything but bootsrapping. I've already said a few things about the Stratego compiler hacking. Since it takes ~3-4 hours for a full build of the Stratego compiler in the Delft buildfarm, I've had a couple of other projects to dive into in parallell. One of these has been the porting of Eclipse IMP from Eclipse 3.2 to 3.3.

In short, IMP is an IDE generator based on Eclipse. It provides set of plugins and wizards that makes the development of programming language environments (a lot) easier. The basic workflow when building an IDE for you favourite language with IMP is, (1) provide a grammar defined using the LPG grammar language, (2) use the IMP-provided wizards inside Eclipse to generate things like syntax highlighting support, outline support, code folding support, templates, text hovers, etc, then (3) fill in the skeletons provided by the generator. My personal view (subject to change without warning) of the generated code is that it's a guide to which parts of the Eclipse framework you need to extend in order to provide a given piece of functionality. Sort of a little helpful gnome pointing you in the right direction. In some cases, the generated code will actually do all you want, but more often than not, you will want to go beyond it.

That was the backgrounder on IMP. A major drawback of the current IMP releases is that they will only work on 3.2. Oh, and, of course, that IMP requires IMP to build IMP. Getting this beast ported to 3.3 wasn't as straightforward as I'd hoped. It took a few iterations. The first was getting it to build properly without any problems on my plain 3.2 installation. That took me several days. All kinds of subtle bugs surfaced, presumably because I have a different set of development habits than the IMPers.

Once those were patched and fixed upstream, I managed to bootstrap my first version on 3.2. An ensuing battle with race conditions in the startup code of various plug-ins followed. I hate static initializers, but apparently not everybody does. In a multi-plugin architecture where the order to plugin loading is not guaranteed, I cannot see how you can safely assume the order of static initializers across plugins, but those questions are not for me to ponder. I ripped them out, and replaced them with lazy initializers as far as possible, and that worked wonders. With that hurdle out of the way, it was all down hill: a couple of internal JFace and JDT classes had changed locations and APIs between 3.2 and 3.3, but it was quick enough to rewrite the offending code (another reason why depending on internal APIs is a bitch, though I realize that the features in question could not have been provided without doing so).

It's a huge disappointment to realize that my patches are only a couple of hundreds of lines. I felt like I had to rewrite the world, at places... Anyway, here's hoping to its inclusion in one of the pending releases. I've updated our sdf2imp tool to use the 3.3-based IMP, so we're already seeing a return on my investment:)

Due to professoral scheduling constraints, I've had to move the Spoofax seminar. New time, same place:

Time: Tuesday 8th of May, 14:15.
Location: Lunch room, 3rd floor.

As part of the PhD formalities, I'm required to hold a seminar on a topic of my own choosing. It's supposed to demonstrate to willing (or not-so-willing) listeners that I know how to talk scientifically and intelligibly about at least one subject.

I decided to present the Eclipse plugin for Stratego I've been hacking on for the last two years. It's becoming a rather interesting development environment, and I suspect that people with an interest in computer languages and/or development environments might actually find some of the material interesting.

Everybody, even (perhaps especially!) non-university folks are of course welcome to drop in, if they have the time.

Software development is expensive primarily because of the associated
maintenance cost; recent estimates suggest that about 70-90% of the
total cost of a software product is due to maintenance. It is therefore
desirable to automate as tasks as possible by supplying convenient and
powerful maintenance tools to the developers. This automation requires
the construction of software that analyses and transforms
other software.

In this talk, I will demonstrate Spoofax, an interactive program
transformation environment based on Eclipse. This environment supports
the development of stand-alone software transformation programs using
the Stratego software transformation language and the XT toolkit of
transformation components. I will demonstrate the applicability of
Spoofax and Stratego/XT through a series of transformation programs for
Java code.

Time: 14:15, Tuesday 8th of May (Updated Again)
Location: Lunchroom, 3rd Floor, Informatikkblokken, Høyteknologisenteret i Bergen
Cookies: Yes

If you're vaguely interested in dropping in, but don't know exactly where to go, contact me or drop by room 4152A (my office) in the fourth floor in the Informatics-wing of HIB around 14:05, and I'll guide you.

Eelco and I have a second paper at the LDTA workshop this year -- a tool description paper about Spoofax. The paper is very space-constrained, so we dropped the abstract, but if we had one, it would read like this:

Many programmable software transformation systems are based around novel domain-specific languages, with a long history of development and successful deployment. Despite their reasonable maturity and applicability, these systems are often discarded as esoteric research prototypes partly because the languages are frequently based on less familiar programming paradigms such as term and graph rewriting or logic programming, and partly because modern development environments are rarely found for these systems. The basic and expected interactive development aids such as source code navigation, searching, content completion, real-time syntax highlighting and error checking, are rarely available to developers of transformation code.

In this system description paper, we introduce Spoofax, an interactive development environment based on Eclipse for developing program analyses and transformations with Stratego/XT -- a language and toolset for developing stand-alone software transformation systems based on formal language descriptions. Spoofax provides the aids mentioned above, in addition to a code outliner and incremental building of projects, and thus significantly eases the development of language processing tools using Stratego/XT. Furthermore, Spoofax is extensible with scripts written in Stratego that are executed within Eclipse, and allow live analyses and transformations of the code under development.

(pdf, bib)

There is already a website for Spoofax, www.spoofax.org, but it's hardly inviting and informative. I have a new one in SVN, but as always, I tend to spend my time hacking code instead of making releases and web pages. I'll try to remedy that very soon.

I have just uploaded the 0.3.11 release, which in reality is a 0.4.0 pre-release, to the update site. I encourage everybody who uses the current version, even 0.3.0, to give 0.3.11 a whirl.

New in 0.3.11:

• Alt+D pops up a list of all definitions (constructors, strategies, rules) which are visible inside the current module. The calculation does not take into account dynamic rule scopes or local definition scopes. The list can interactively be searched, using * and ? as wildcards.
• Alt+M pops up a list of all modules visible from the current module, also searchable.
• F3 (go to definition) now also suggests constructors, and will no longer be confused by rules and strategies with parameters.
• Source code navigation is generally more accurate than before, and seems to work reliably inside the the Stratego standard library. Just from playing around, I have discovered many cases where the imports are underspecified: strategies are referred to that are never imported.
• More xDoc (all?) code is syntax highlighted.
• rec, let and in are syntax highlighted.

With any luck, I have kept all the old bugs that people have come to appreciate and enjoy, while at the same time introducing an all new flora that may appeal to newcomes.

I decided to forgo my friday night clubbing and rather spend the time at the office in solitude with a bottle of water, some microwave food and Eclipse windows floating feverishly about on my dualhead setup (I don't have an extra monitor at home).

The end result was another release of the Spoofax editor, codenamed Obelix. The name is apt; Obelix is 322% larger than Asterix (0.2.0) and weighs in at around 87KB. In addition to more crud, more features were also added:

• An outline view for Stratego files

  At the present time, the outliner will only update itself whenever you save. I have not added an autotimeout for reparsing, but this shouldn't be all that difficult.

  By clicking on the items in the outline view, the focus of the connected editor will be changed automatically, and the area in the source file associated with the declaration will be marked in the left hand side margin of the editor. Unfortunately, due to parsing inaccuracies, only the area corresponding to the declaration of the identifier will be marked, not all of its body too. I hope we can fix this soon.

  The information from provided by the outline is kept in the StrategoModel class, which I hope will evolve into a proper AST as time goes by and we improve our parser.
  

• Content assistance for strategies, rules and constructors
  

  

  Whenever you press Ctrl-Space, the content assistant will be run. It will consider the location you are in the document, parse leftwards and see if it can find something resembling strategy, rule or constructor identifier.

  When the scanner finds an identifier, it will assume it's a rule or strategy, and will show you all potential expansions for the prefix it found. By selecting one, it will insert the selection and move the cursor to after the completion. If you complete a higher-order strategy or rule, it will add a set of parentheses and place the cursor inside these.

  If, while scanning leftwards, it finds a !, it will assume you want a list of constructors.

  By typing ! or <, then waiting 500msec, the editor will automatically pop up a list of constructors or rule/strategy-definitions, respectively, all by itself.

• Preference pages for syntax highlighting
  

  

  Now the editor does not have to be restarted after changing the settings anymore. They are applied instantly.

• A nifty realtime parenthesis matcher
  

The parenthesis matcher keeps track of the kinds of parenthesis used, and if they don't match, it will signal a mismatch by a red (customizable) match cursor.

The interesting part of this -- content assist and outliner -- was made possible thanks to the new "outline" parser which is somewhat context sensitive. It skimps along the document trying to pick out anything it can reasonably detect, and places it into the StrategoModel. There's a long way left before we have 100% correct parsing. In particular, the unrestricted syntax embedding allowed in Stratego source code is of course not generally solvable. But as we approach there, we will have increasingly better completion and navigation.

A few words of acknowledgement are definitely in order. Inspiration by Martin helped a lot. On the technical side, looking at the SchemeWay project by Dominique Boucher was extremely helpful. My parenthesis matcher is heavily inspired by his. Also, I can heartily recommend the JavaEditor example in the Examples SDK to nail the basics of Eclipse editor development.

In an attempt at returning to normalcy after over a week of nothing but frantic documentation writing (in the form of an article), I have spent this week looking at module systems, modularization techniques and known methods for exposing variation points in these.

Apart from reading papers, I decided to experiment with extending frameworks, and I also had another itch to scratch. The result is a very rudimentary syntax highlighting text editor plugin for Eclipse, for the Stratego language.

For now, I've put it up on the temporary front page of spoofax.org, and I even provided a screenshot.

If you run Eclipse 3.0 or newer, you can download and install it using the Update Manager, as usual.

A lot of things are still missing, the most important (to me) of which are: context assist for strategies, rules, variables and constructors; outlining of strategy and rule definitions; a sensible indentation strategy; and semantic highlighting of variables in rule conditions and strategy expressions.

I've spent the last few months browsing for and testing useful Eclipse plugins.

If one were to divide the plugins into commerical and open-sourced, there's one striking difference between the two groups: As usual, it is the case the commercial plugins have a better finish, and seem more robust, provided they actually start. With the open-sourced ones, it is usually the case that they start, but invariably have many minor and major rough edges that make them tricky to use.

I'd like to mention a handful of plugins that are of special note.