This last week, I spent some free cycles hacking together a small project instantiation tool for Stratego/XT. It makes setting up a fresh Stratego project really simple by automatically populating the project space with a default directory layout, build system files and some minimal program and syntax samples.

To create a project p0, all you have to do is:

$ crap --new-project p0

This creates all the files necessary for a complete GNU Autotools-based build system, including a sample Stratego program (src/xmpl.str):

p0/
   Makefile.am
   README.Developer
   README
   AUTHORS
   bootstrap
   p0.spec.in
   NEWS
   p0.pc.in
   configure.ac
   ChangeLog
   xmpl/
        Makefile.am
   syn/
       Makefile.am
   tests/
         Makefile.am
   src/
       Makefile.am
       xmpl.str

Once this is done, you can configure and compile the project,

$ ./bootstrap
$ ./configure
$ make all

install it,

$ make install

and even run the example transformation program:

$ echo "foo" | /usr/local/bin/xmpl
"Hello, World!"

The example program expects an input on stdin [or in a file specified by the -i switch], and will always produce the output string "Hello, World!".

The crap tool is part of the strategoxt-utils package. You can also download a stand-alone snapshot of crap.

More comprehensive documentation is available in the wiki. The tool is still very rough, so any suggestions for improvements and bug reports are very welcome.

I finally figured out how to add proper navigation history support to Spoofax today. This one has been bugging me for quite some time. I remember spending far too much time diving through the documentation with the hopes of figuring out how this should be done properly. No luck.

Today I had a flash of inspiration, so I dug into the JDT code base. That code seemed to solve the same problem in a very complicated way, so I didn't want to copy their approach outright. Stymied, I started tracing exactly what happens with the navigation history when positions are placed into it. After a bit of fiddling around, I figured out that when I move the cursor, I should mark the position both before and after the cursor/focus moves to get the behaviour of JDT (which I tried to emulate). I've always only tried saving the editor location state either before I changed it, or afterwards. I also tried all kinds of alternative calls on the EditorPart hierarchy in vain. I now use ITextEditor.setHighlightRange() which appears to do the job, provided I call markInNavigatorHistory() "properly".

Anyway, the lesson is simple: if you call AbstractTextEditor.markInNavigationHistory(), remember to do it twice -- once before you change the editor/focus and once afterwards.

Due to professoral scheduling constraints, I've had to move the Spoofax seminar. New time, same place:

Time: Tuesday 8th of May, 14:15.
Location: Lunch room, 3rd floor.

As part of the PhD formalities, I'm required to hold a seminar on a topic of my own choosing. It's supposed to demonstrate to willing (or not-so-willing) listeners that I know how to talk scientifically and intelligibly about at least one subject.

I decided to present the Eclipse plugin for Stratego I've been hacking on for the last two years. It's becoming a rather interesting development environment, and I suspect that people with an interest in computer languages and/or development environments might actually find some of the material interesting.

Everybody, even (perhaps especially!) non-university folks are of course welcome to drop in, if they have the time.

Software development is expensive primarily because of the associated
maintenance cost; recent estimates suggest that about 70-90% of the
total cost of a software product is due to maintenance. It is therefore
desirable to automate as tasks as possible by supplying convenient and
powerful maintenance tools to the developers. This automation requires
the construction of software that analyses and transforms
other software.

In this talk, I will demonstrate Spoofax, an interactive program
transformation environment based on Eclipse. This environment supports
the development of stand-alone software transformation programs using
the Stratego software transformation language and the XT toolkit of
transformation components. I will demonstrate the applicability of
Spoofax and Stratego/XT through a series of transformation programs for
Java code.

Time: 14:15, Tuesday 8th of May (Updated Again)
Location: Lunchroom, 3rd Floor, Informatikkblokken, Høyteknologisenteret i Bergen
Cookies: Yes

If you're vaguely interested in dropping in, but don't know exactly where to go, contact me or drop by room 4152A (my office) in the fourth floor in the Informatics-wing of HIB around 14:05, and I'll guide you.

Eelco and I have a second paper at the LDTA workshop this year -- a tool description paper about Spoofax. The paper is very space-constrained, so we dropped the abstract, but if we had one, it would read like this:

Many programmable software transformation systems are based around novel domain-specific languages, with a long history of development and successful deployment. Despite their reasonable maturity and applicability, these systems are often discarded as esoteric research prototypes partly because the languages are frequently based on less familiar programming paradigms such as term and graph rewriting or logic programming, and partly because modern development environments are rarely found for these systems. The basic and expected interactive development aids such as source code navigation, searching, content completion, real-time syntax highlighting and error checking, are rarely available to developers of transformation code.

In this system description paper, we introduce Spoofax, an interactive development environment based on Eclipse for developing program analyses and transformations with Stratego/XT -- a language and toolset for developing stand-alone software transformation systems based on formal language descriptions. Spoofax provides the aids mentioned above, in addition to a code outliner and incremental building of projects, and thus significantly eases the development of language processing tools using Stratego/XT. Furthermore, Spoofax is extensible with scripts written in Stratego that are executed within Eclipse, and allow live analyses and transformations of the code under development.

(pdf, bib)

There is already a website for Spoofax, www.spoofax.org, but it's hardly inviting and informative. I have a new one in SVN, but as always, I tend to spend my time hacking code instead of making releases and web pages. I'll try to remedy that very soon.

I have just uploaded the 0.3.11 release, which in reality is a 0.4.0 pre-release, to the update site. I encourage everybody who uses the current version, even 0.3.0, to give 0.3.11 a whirl.

New in 0.3.11:

• Alt+D pops up a list of all definitions (constructors, strategies, rules) which are visible inside the current module. The calculation does not take into account dynamic rule scopes or local definition scopes. The list can interactively be searched, using * and ? as wildcards.
• Alt+M pops up a list of all modules visible from the current module, also searchable.
• F3 (go to definition) now also suggests constructors, and will no longer be confused by rules and strategies with parameters.
• Source code navigation is generally more accurate than before, and seems to work reliably inside the the Stratego standard library. Just from playing around, I have discovered many cases where the imports are underspecified: strategies are referred to that are never imported.
• More xDoc (all?) code is syntax highlighted.
• rec, let and in are syntax highlighted.

With any luck, I have kept all the old bugs that people have come to appreciate and enjoy, while at the same time introducing an all new flora that may appeal to newcomes.

I decided to forgo my friday night clubbing and rather spend the time at the office in solitude with a bottle of water, some microwave food and Eclipse windows floating feverishly about on my dualhead setup (I don't have an extra monitor at home).

The end result was another release of the Spoofax editor, codenamed Obelix. The name is apt; Obelix is 322% larger than Asterix (0.2.0) and weighs in at around 87KB. In addition to more crud, more features were also added:

• An outline view for Stratego files

  At the present time, the outliner will only update itself whenever you save. I have not added an autotimeout for reparsing, but this shouldn't be all that difficult.

  By clicking on the items in the outline view, the focus of the connected editor will be changed automatically, and the area in the source file associated with the declaration will be marked in the left hand side margin of the editor. Unfortunately, due to parsing inaccuracies, only the area corresponding to the declaration of the identifier will be marked, not all of its body too. I hope we can fix this soon.

  The information from provided by the outline is kept in the StrategoModel class, which I hope will evolve into a proper AST as time goes by and we improve our parser.
  

• Content assistance for strategies, rules and constructors
  

  

  Whenever you press Ctrl-Space, the content assistant will be run. It will consider the location you are in the document, parse leftwards and see if it can find something resembling strategy, rule or constructor identifier.

  When the scanner finds an identifier, it will assume it's a rule or strategy, and will show you all potential expansions for the prefix it found. By selecting one, it will insert the selection and move the cursor to after the completion. If you complete a higher-order strategy or rule, it will add a set of parentheses and place the cursor inside these.

  If, while scanning leftwards, it finds a !, it will assume you want a list of constructors.

  By typing ! or <, then waiting 500msec, the editor will automatically pop up a list of constructors or rule/strategy-definitions, respectively, all by itself.

• Preference pages for syntax highlighting
  

  

  Now the editor does not have to be restarted after changing the settings anymore. They are applied instantly.

• A nifty realtime parenthesis matcher
  

The parenthesis matcher keeps track of the kinds of parenthesis used, and if they don't match, it will signal a mismatch by a red (customizable) match cursor.

The interesting part of this -- content assist and outliner -- was made possible thanks to the new "outline" parser which is somewhat context sensitive. It skimps along the document trying to pick out anything it can reasonably detect, and places it into the StrategoModel. There's a long way left before we have 100% correct parsing. In particular, the unrestricted syntax embedding allowed in Stratego source code is of course not generally solvable. But as we approach there, we will have increasingly better completion and navigation.

A few words of acknowledgement are definitely in order. Inspiration by Martin helped a lot. On the technical side, looking at the SchemeWay project by Dominique Boucher was extremely helpful. My parenthesis matcher is heavily inspired by his. Also, I can heartily recommend the JavaEditor example in the Examples SDK to nail the basics of Eclipse editor development.

I released an update to the Stratego/XT editor plugin part of Spoofax today. This new version adds simple syntax highlighting support for SDF files.

I am still in the process of rewriting the Stratego parser to enable code completion and outlines. Further, I've started thinking about how to couple the editor with the compiler to map back Stratego compiler errors. I think this would easy as π if strc were to emit the errors as an ATerm.

All of this is slow going, as I have more pressing matters to finish.

To test the plugin provide the URL
http://www.ii.uib.no/~karltk/spoofax/
to the Eclipse Update Manager.

An enticing screenshot:

In an attempt at returning to normalcy after over a week of nothing but frantic documentation writing (in the form of an article), I have spent this week looking at module systems, modularization techniques and known methods for exposing variation points in these.

Apart from reading papers, I decided to experiment with extending frameworks, and I also had another itch to scratch. The result is a very rudimentary syntax highlighting text editor plugin for Eclipse, for the Stratego language.

For now, I've put it up on the temporary front page of spoofax.org, and I even provided a screenshot.

If you run Eclipse 3.0 or newer, you can download and install it using the Update Manager, as usual.

A lot of things are still missing, the most important (to me) of which are: context assist for strategies, rules, variables and constructors; outlining of strategy and rule definitions; a sensible indentation strategy; and semantic highlighting of variables in rule conditions and strategy expressions.