Using @shadow

This chapter describes an important new feature that debuted in Leo 4.5 b2: @shadow trees. These trees combine the benefits of @auto, @file and @nosent trees:

  • The (public) files created by @shadow trees contain no sentinels, but
  • Leo is able to update @shadow trees in the Leo outline based on changes made to public files outside of Leo.

@shadow trees are often useful for studying or editing source files from projects that don’t use Leo. In such situations, it is convenient to import the @shadow tree from the (public) sources. As discussed below, Leo can import @shadow trees automatically, using the same algorithms used by @auto trees.

The crucial ideas and algorithms underlying @shadow trees are the invention of Bernhard Mulder.

Overview

Using @shadow trees is the best choice when you want to have the full power of Leo’s outlines, but wish to retain the source files in their original format, without Leo sentinels (markup) in comments in the source file.

Leo’s @file trees create external files containing comments called sentinels. These sentinel lines allow Leo to recreate the outlines structure of @file trees. Alas, many people and organizations find these added sentinel lines unacceptable. @nosent nodes create external files without sentinels, but at a cost: Leo can not update @nosent trees when the corresponding external file is changed outside of Leo.

@shadow trees provide a way around this dilemma. When Leo saves an @shadow tree, it saves two copies of the tree: a public file without sentinels, and a private file containing sentinels. Using Bernhard Mulder’s brilliant update algorithm, Leo is able to update @shadow trees in the Leo outline based solely on changes to public files.

Leo writes private files to a subfolder of the folder containing the public file: by default this folder is called .leo_shadow. You can change the name of this folder using the @string shadow_subdir setting. Note that private files need not be known to source code control systems such as bzr or cvs.

That’s almost all there is to it. The following sections discuss important details:

  • How to create @shadow trees.
  • How @shadow works.
  • Why the update algorithm is sound.

Creating @shadow trees

The first step in creating an @shadow tree is to create a node whose headline is @shadow <filename>.

Thus, you can create an @shadow node and save your outline, regardless of whether the original file exists. The next time Leo reads the @shadow node, Leo will create the entire @shadow tree using the same logic as for @auto trees. You can cause Leo to read the @shadow node in two ways: 1) by closing and reloading the Leo outline or 2) by selecting the @shadow node and executing the File:Read/Write:Read @shadow Node command.

Important: Leo imports the private file into the @shadow tree only if

  1. the public file exists and
  2. the private file does not exist.

Thus, Leo will import code into each @shadow node at most once. After the first import, updates are made using the update algorithm.

Note: just as for @auto, Leo will never read (import) or write an @shadow tree if the @shadow node is under the influence of an @ignore directive.

What the update algorithm does

Suppose our @shadow tree is @shadow a.py. When Leo writes this tree it creates a public file, a.py, and a private file, .leo_shadow/xa.p (or just xa.p for short). Public files might can committed to a source code control system such as cvs or bzr. Private files should not be known to cvs or bzr.

Now suppose a.py has been changed outside of Leo, say as the result of a bzr merge. The corresponding private file, xa.p, will not have been changed. (Private files should never change outside of Leo.

When Leo reads the new (and possibly updated) public file it does the following:

  1. Recreates the old public file by removing sentinels from the (unchanged!) private file.
  2. Creates a set of diffs between the old and new public files.
  3. Uses the diffs to create a new version of the private file.
  4. Creates the @shadow tree using the new private file.

Important: The update algorithm never changes sentinels. This means that the update algorithm never inserts or deletes nodes. The user is responsible for creating nodes to hold new lines, or for deleting nodes that become empty as the result of deleting lines.

Step 3 is the clever part. To see all the details of how the algorithm works, please study the x.propagate_changed_lines method in leoShadow.py. This code is heavily commented.

Aha: boundary cases don’t matter

There are several boundary cases that the update algorithm can not resolve. For example, if a line is inserted at the boundary between nodes, the updated algorithm can not determine whether the line should be inserted at the end of one node of the start of the next node.

Happily, the inability of the update algorithm to distinguish between these two cases does not matter, for three very important reasons:

  1. No matter which choice is made, the public file that results is the same. The choice doesn’t matter, so the update algorithm is completely and absolutely safe.
  2. Leo reports any nodes that were changed as the result of the update algorithm. In essence, these reports are exactly the same as the reports Leo makes when @file trees were changed as the result of changes made externally to Leo. It is as easy for the user to review changes to @shadow trees as it is to review changes to @thin or @file trees.
  3. Suppose the user moves a line from the end of one node to the beginning of the following node, or vice versa. Once the user saves the file, the private file records the location of the moved line. The next time the user reads the @shadow file, the line will not be subject to the update algorithm because the line has not been changed externally. The location of the line (on the boundary) will be completely determined and it will never need to be moved across the boundary.

Understanding these three reasons finally convinced me that @shadow could be made to work reliably.

Previous topic

Debugging with Leo

Next topic

Leo and Emacs