In contrast to the modern text editors which could be called language-aware, you might call structured editors language-driven. This expresses a more strict and constrained approach to editing, where each change is carried out on language constructs and not on a list of text lines. The syntax and grammar of the language define what language construct can be inserted at the current insertion point, and the editor provides choice for the user.
Unlike plain text editors, a structured editor cannot represent or operate on a part of a language construct – it operates on AST elements as a whole. For instance, in plain text you can have an incomplete construct with an opening brace but without the closing brace. A structured editor doesn’t need braces at all. In text, you can delete part of a concept by deleting incomplete selection. In a structured editor, you can only delete a whole concept.
The atomicity of editor operations guarantees that language constructs are being added, moved, renamed or removed as a whole. The typedness of language constructs would only allow correct embedding of concepts – each concept can only host those concepts which are allowed by the language grammar. For example, you wouldn’t be able to move a method to a namespace level, or a foreach statement to a class, or declare an implementation on an interface method.
Each change to the data structure is of transactional nature and is language-aware. For instance, one would speak of changes in terms of “a class was inserted” instead of “text was inserted”. It is easier to track all the changes in an Undo/Redo buffer and to compose multiple changes to a transaction.
Most syntax errors are simply impossible within a structured editor, so the program being edited is always syntactically correct (with minor exceptions). For example, representing hierarchical concepts as nested blocks (and not with a set of paired delimiters) guarantees that the curly braces and #region directives will always be balanced. Curly braces earlier used to delimit tree nodes become now unnecessary.
When you are inside a block, only blocks of certain type can be inserted, dependent on language syntax. You can continuously move your caret through your code as if it were plain text. All the blocks are formatted automatically.
Few remaining errors, which are not catched by the structure itself, could be easily found and highlighted by using a tree checker visitor. The implementation would be simpler and wouldn’t theoretically even require a scanner or a parser for this purpose. Thus the editor can automatically ensure syntax correctness by its language-awareness.
Since a structured editor has more ready knowledge about the program being edited and the semantics of the language constructs, it can more easily offer more intelligent user feedback on the program and offer only valid choices based on the current context. The context itself can only be a correct language construct or a correct insertion point where a language construct can be created.
A structured editor can also reduce the number of required primitive user input operations (key presses, mouse clicks and moves). For example, it is sufficient to select a block and hit the [Delete] key to remove a concept. In a text editor, one would first have to select the text which corresponds to the concept and only after that press [Delete]. When a concept spans more than one screen (for example, a big class), it is still easy to select and delete it as a whole in the structured editor, whereas selecting the class in a text editor would require scrolling.
Each block can be selected, collapsed, commented, cut or copied to clipboard, moved into another position in the parent block or into another block with just one click or key press. Entire block hierarchies are selectable just like usual blocks.
With a structured editor the user doesn’t have to worry about indentation, semicolons, curly braces etc. The editor takes care of the syntax so that the user can concentrate on editing the program and not on remembering the syntax for a language construct. The code is being automatically “formatted” as it is typed in, because it is actually displayed from a live AST, which is always correctly “formatted”. The user mostly enters important content and not irrelevant formatting symbols.
Structured editors actually open wide possibilities for usability research. It is a complex task to gather statistical information about how different developers perform elementary typing operations and what elementary user input actions are required (mouse and key clicks, mouse moves, pen gestures). One thing is sure: developers don’t actually notice, how exactly they type – they just type without thinking about it and develop some sort of reflexes when working with an editor for a long time. Structured editors could substantially shorten required user input actions when compared to text editors. This will most probably be a problem for those who are already using text editors for a long time, but newcomers will probably appreciate it, especially because they don’t really have to remember the syntax (“were there semicolons inside a for loop or commas?”).
Context help for an editor could provide important hints for the user based on the current selection and caret position. Implementing context help would be much simpler too – one could simply declaratively add help pages to the syntax constructs, without the need to parse the source code to determine what construct is currently near the caret.
An intelligent scrollbar could allow easy and powerful code navigation. The scrollbar could contain miniature portions of the code document. This approach is being explored by the Human Interactions in Programming group [HIP] inside Microsoft Research – in a project called “Code Thumbnails”.
A so-called “breadcrumb” control could be provided for improved navigation experience. When a block is active (focused), an extra control could show a horizontal list of all parent blocks as a chain. Each link of the chain could actually be a hyperlink to the corresponding parent block, or a combo-box with the drop-down list of all other siblings at this parenting level.
Another possible future direction for usability research is key stroke prediction based on statistics – gathering and accumulating typing experience from users all around the world could provide some AI-driven preselection what construct is going to be used next, just like IntelliSense in Visual Studio 2005 remembers the last used type and member.
For those who are sceptical about the usability of a structured editor because it will be so unusual and difficult to use, Wesner Moise gives a probable answer to this in [WesnerM1]:
… Such graphical editors won't actually appear much different from text editors; they may even look the same as text editors, with some differences.”
Thus structured editors could mimic traditional text editors to provide familiarity but still retain the advantages of structured editing.
Another good thing about structured editors is that they are very easily skinnable, because the content is decoupled from the presentation. Just like a CSS stylesheet can provide a totally different look for an XHTML document, one could easily change the presentation of how language constructs are visualized on the screen. It is the content and the structure of that content that matters.
These are the reasons why XHTML + CSS were developed to replace old HTML where the presentation was mixed with the content. Another example for this development are styles in word processors, which are definitely better than inline formatting scattered all over the document in multiple places.
When the presentation can be adjusted independently of content, preserving team’s coding guidelines becomes a snap. One could virtually “edit” the teams coding guidelines in a special domain-specific editor, for example in a graphical design software with gradients and visual effects, and the entire source code would be automatically displayed with this style on the fly. Or, every developer could even have own coding guidelines (style library) – and have all the code automatically displayed with her/his favorite style.
When you open the program in another IDE with different settings, the code could be displayed using these settings automatically, without the need to manually reformat it. Reformatting code would become unnecessary. Code could be automatically formatted for printing based on one of printing templates.
Again, this is another manifestation of the MVC principle: one could easily switch views or use several different synchronized views at the same time, as long as the model (content) is clearly separated from the presentation.
Another nice feature of structured editors is the possibility to collapse everything, to only show the currently relevant code. This could go much further than by text editors – one could even apply custom sort or filter to display only relevant things in the desired order. The order in which the declarations are physically stored in the file would become separated from the logical order in which they are presented to the user. Different developers could even have the same code sorted as they want it.
This is a common reason for disagreement in teams about coding guidelines – some prefer to have all the fields at the beginning of the class, and some prefer to have each backing field attached to its corresponding wrapping property/getter/setter.
Structured editors are also well suited for embedding other mini-languages (DSL) within the main (host) language. Just as LINQ features are a sort of “embedded SQL/XML” in C# 3.0, structured editors could easily embed custom DSL, for example mathematical notation for expressions, with the square root sign and vertical fraction notation.
An editor could contain custom interactive controls to improve the editing experience for specialized content, for example a color-choice popup box where a variable of type Color is required.
Other visual formatting could be added as necessary. For example, custom lines, rules and delimiters could be used instead of ASCII-art style comments:
// = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Lutz Roeder gives insightful examples of such enrichments in [LutzR1].
A structured editor could simplify and automate many programmer’s tasks that now have to be carried out manually. Backing fields for properties can be entered and displayed concisely (C# 3.0 now does this with its auto-implemented properties).
A structured editor could provide a platform for extending the language by defining some equivalent of macros. One could define a new type of a language construct externally and then just plug this type into the grammar of the editor. Thus very concise and expressive shortcuts for long pieces of code could be defined, which could be expanded into longer code sections during the code generation. The difference between such macros and, say, C #define macros is that the structured macros (unlike C macros) would be language-aware, and thus all the type-checking and verification can actually take place. The problem with C macros was exactly the lack of language-awareness – the substitution took place before parsing, at the preprocessing stage. With language-aware macros the substitution takes place after parsing – language constructs are being inserted into the existing AST, thus allowing all checking visitors to run on the final tree. Moreover, the editor’s grammar and type-checking facilities could only allow inserting macro instances at correct places in the tree. This eliminates the problems of text-based macros and preprocessing.
The [Nemerle] programming language gives a great example of language-aware macros in a text-based language.
The structured, factored nature of the editor would allow to more easily extend the language without actually extending the language. Plugging in new language concepts probably wouldn’t even require recompiling the editor. A new ecosystem for “language plug-ins” could arise, which would allow each software vendor to develop tools for the product parallel to the product itself, in the sense of Microsoft’s Software Factories initiative ([SoftFact]).
A great research about extensibility of structured editors is given by Intentional Programming ([IntentSoft], [Simonyi1]).
An IDE built around AST structures and backed by a structured editor that directly operates on this structure could really simplify the architecture and implementation of many components, such as code completion, refactoring, navigation, debugger, edit-and-continue, etc. The thing is, it is easier to make the IDE intelligent if there is no need to constantly round-trip between text and AST. A good example is extracting a method, which would be simpler to implement in comparison with some current implementations, where a lot of effort has to be put into the round-tripping. When some code has to be inserted into the program with a text editor, a major problem is finding the correct place to insert and take care of the formatting. A structured editor takes care of the formatting.
An important advantage in the implementation is eliminating the need for a background compiler. When the AST is kept up-to-date by the editor itself, there is no need to reparse code in regular intervals in a background thread. When there is no background thread, there is no danger that the program information would get out of date, which can be the case with background compilation. Implementing a background compiler is not an easy task and freeing the developers from the need to implement one could simplify the overall architecture of the IDE and keep it more robust and consistent.
It may well be the fact that the performance of a structured editor (especially when doing complex language-aware operations such as code completion or refactoring) would far surpass that of a text editor. Resolving and reparsing is a common bottleneck of traditional language services, and a structured approach could handle this elegantly.
When the source code is stored as text, the internal representation isn’t stored with it – it is being reconstructed every time when the source is loaded into the IDE. That’s why the changes are expressed in terms of changed lines in version control systems. Moreover, current version control systems do not provide feedback about, for instance, how many classes were changed, how many methods have been added/removed/moved/edited, etc.
When the parsed and resolved AST is stored in a versioned and transactional database, changes could be registered in fine-granular, language-aware manner: classes added, methods edited, variable renamed, etc.
Thus the editor (“View”) could reside on the client and the code being edited (“Model”) could be inside a database repository on a (possibly remote) server. This could even potentially allow several people to edit the same program at the same time (while viewing updates in realtime), which is more fine-granular than current version control systems.
Branching and merging operations could be carried out more easily and in terms of changes to language constructs. Because the order of declarations on a class or namespace level is visible to the system, moving a method within the same class wouldn’t even be considered a change for the end-user – the users who carry out forward- and reverse-integration processes now can concentrate on the meaning of the program, instead of formatting.
Another advantage of language-aware source code repositories that store AST in a database is a requirement, that every code being checked in must be at least syntactically correct. When implementing such a system, it surely would be possible to only allow to check in code that compiles, because the system would understand the code which is being checked in.
A language-aware version control system can prevent breaking builds.
Moreover, the check in process as such, could be replaced in the future by more fine-granular “transactions on code” – literally each change would be registered in the repository as a transaction and one could group changes manually to the desired granularity level, thus grouping changes to larger entities (for example, one day’s work). The change groups could be grouped too (design pattern Composite) to encapsulate features, milestones, products, etc.
A web-service based, programming language independent source code repository would allow developers to access shared code from desktop computers or mobile devices. The code can be automatically formatted and represented in any preferred way on the client (by the editor), whereas the repository is formatting-agnostic.
A lot of insight about storing source code in some intermediate representation (e.g. a relational database) is given in [SCID]. A big application of storing source code in database could be static analysis tools.
Static analysis of source code and source code querying are becoming more and more popular nowadays. Tools like [FxCop], [NDepend], [Semmle] and [NStatic] either store the parsed code in a relational database, parse the code on the fly or analyze assemblies/byte-code. Either way, they operate on the AST, and if the AST exists during code editing and is always up-to-date even without the background compiler, such tools could greatly benefit by eliminating the need for a custom parser or even provide a real-time “as-you-type” analysis experience, where possible.
A good thing about structured editing is that it is more explicit about the language constructs being edited. A structured editor better reflects the inner structure of a program, how it is seen by a compiler.
Currently, the first thing beginner programmers learn is the textual syntax of a programming language, not the language constructs themselves. A common beginners question when learning programming is “what [language construct] can I actually insert at the current position”? Other beginners problems are, for example, remembering the right keyword, spelling and order of arguments, typing and formatting, matching curly braces, indenting blocks with tabs etc.
These questions could be better answered by a structured editor, where syntax constructs and their embeddings can be more clearly visualized, and an alternative list would only show possible constructs.
Also, programmers use different programming languages. In particular, the Microsoft .NET Framework supports many languages in a single environment. Understanding code, written in another .NET language, is for many an obstacle. However it is often the case that the languages differ mostly by syntax and it is the syntax that prevents a C# developer to immediately recognize a constructor in “Sub New ... End Sub”. The structured approach allows to at least partly erase the boundaries between different .NET programming languages and those who use them.
Previous: 1.3. Principles of plain text editors