Chapter 4: Technical information

The following information describes the Yodl package from the point of the system administrator. Issues such as the installation of the package are addressed here.

4.1: Obtaining and installing Yodl

The Yodl program and the distributed macro package can be obtained at the ftp site ftp.lilypond.org in the directory pub/yodl/development. Look for a file called yodl-X.Y.Z.tar.gz, where X.Y.Z is the highest version number. This is a gzipped archive containing all sources, documentation and macro files.

4.1.1: Configuring the yodl program

Once you unpack the archive, configure the sourcetree with a command that looks remotely like

        configure    # Check out the bin/set-yo.sh script
        make
        make install

The configuration process is quite versatile; it should run flawlessly to detect your system and its defaults. You may alter various settings, see configure --help.

4.1.2: Installing the yodl program

Once configured, type

    make all

to build everything. If everything went ok, you can do

    make install

to install it. The executable, which is built as src/out/yodl is created and copied to a system-wide program directory. The macro package from macros/ is also placed in a public directory, which is /usr/local/share/yodl by default (you can change most directory names in the configure process). Furthermore, postprocessors and a number of shell scripts (drivers of the yodl program) are copied to your programs directory.

4.1.2.1: Prerequisites for the installation

To successfully build and install the Yodl package, the following tools must be present:

  • A C compiler and run-time environment. A POSIX-compliant compiler, libraries and set of header files should work without problems. The GNU gcc compiler 2.7 and above should work without a flaw.

  • GNU make

  • Typical building programs, such as, install. Most Unixes will have these.

  • /bin/sh: a POSIX-compliant shell interpreter. The GNU shell interpreter bash works without problems.

  • A number of `standard' tools should be present: sed, grep, etc.. These tools must furthermore include the code generators bison and flex (yacc- and lex lookalikes) to genererate the grammar parsers. The GNU implementations of these tools work like a charm.

  • A command that converts groff input into viewable format. The default setting for this command is troff -Tascii -man.
  • 4.2: Internal workings of the Yodl package

    This section describes the internal workings of the yodl package, for those who are interested (and of course, for the maintainer).

    4.2.1: A single-pass interpreter

    The yodl program is a single-pass interpreter of its input files. The program does not build an internal data representation of the text and commands that it encounters; therefore, all actions must be completed in one pass. The basic way by which yodl does this, is by reading input, taking actions and possibly `pushing back' information on the input.

    This is best illustrated with an example. Given the code,

    DEFINESYMBOL(sym)
    
    IFDEF(sym)
        (Symbol sym is defined!
         DEFINEMACRO(testmac)(0)(Testmac now stands for one thing.)
        )
        (Symbol sym is not defined!
         DEFINEMACRO(testmac)(0)(Testmac now stands for another thing.)
        )
        
    testmac()
    

    yodl will take the following actions:

  • The DEFINESYMBOL command updates an internal table of symbols, no other action is necessary here.

  • When the IFDEF command is parsed, yodl eats up all of the necessary parts from the input: the keyword itself, and three parameter lists. The IFDEF test obviously yields `true', and therefore yodl `pushes back' the true-list

    Symbol sym is defined!
    DEFINEMACRO(testmac)(0)(Testmac now stands for one thing.)
    

  • Next, yodl sees normal text (Symbol sym is etc.). This is sent to the output file.

  • After this, the DEFINEMACRO statement is seen, which defines testmac() as a placeholder for one thing.

  • Finally, testmac() is seen. The parser recognizes a user-defined macro having no arguments, which conforms to the definition. The text

    Testmac now stands for one thing.
    

    is pushed back.

  • Finally, yodl reads this last string from its input and sends it to the output.

  • The most complex part of the program is therefore the tokenizer (lexical analyzer): that part is responsible for matching tokens such as identifiers in the input, and for pushing back text. The pushing back is implemented as a relocatable buffer, which is expanded as text is pushed into it and which shrinks as the lexical analyzer processes input. I already achieved about 100% speed gain by optimizing the tokenizer, but this implementation can still be somewhat of a CPU hog. (But hey, premature optimzation is the root of all evil, says D.E. Knuth. So why optimize, as long as you don't know whether you're being premature about it?)

    The lexical analyzer is furthermore responsible for the line continuation feature (see section 2.2.2) and for the SUBST mechanism (see section 2.3.30). These mechanisms are also implemented with the push-back method.

    4.2.2: Macros of the converters

    The macro files for all converters are (by default) installed to a directory /usr/local/lib/yodl. This is a good place to look if you want to see how a converter works, or if you want to write a new converter.

    The place to start looking is the file shared.yo, which is used by all converters. This file defines most commands that are `global' in the sense that all converters implement them: e.g., bf, em and so on. The implementation is per macro as follows, illustrated for e.g., bf (the actual implementation may be slightly different):

    DEFINEMACRO(bf)(1)(\ 
            whenlatex(latexcommand({\bf )ARG1+latexcommand(}))\ 
            whenhtml(htmlcommand(<strong>)ARG1+htmlcommand(</strong>))\ 
            whenman(mancommand(\fB)ARG1+mancommand(\fP))\ 
            whensgml(sgmlcommand(<bf>)ARG1+sgmlcommand(</bf>))\ 
            whentxt(ARG1))
    

    The macro expands to a set of commands that are active per output format, using latexcommand, htmlcommand etc..

    The file shared.yo is included in its turn by tex.yo, html.yo and so on. These separate files are very short; their function is to:

  • define the conversion type with an appropriate symbol (latex for latex conversions, html for HTML, and so on),

  • activate character tables (when necessary),

  • include shared.dm,

  • define special macros for the particular conversion (when necessary).

  • 4.2.2.1: Adding a new macro

    This section describes how to add a new user-defined macro. For the sake of simplicity, a macro fname(name) is defined to typeset a filename. The macro maps to em() in LaTeX output or to bf() in other formats.

  • The place to insert the macro is lib/yodl/shared.yo.in in the Yodl source tree. This is the `master file' from which both the final macrofile and relevant documentation is created.

  • The macro must appear in alphabetical order in the list of all other macros, this is relevant for the documentation. Hence, the fname macro would appear somewhere near footnote() though before it.

  • The documentation about the hypothetical macro must be set between two tags @startdoc and @enddoc. The make process looks for these tags. The documentation itself must appear as a macro which is (appropriately) called macro, and expects two arguments: the new macro that is being defined, and its description:

    @startdoc
    macro(fname(file))
    (Sets a filename in either boldface or italics,
    depending on the output format.)
    @enddoc
    

  • Following the documentation, the macro itself is defined using whenformat() commands and/or formatcommand():

    
        DEFINEMACRO(fname)(1)(\ 
            whenlatex(em(ARG1))\ 
            whenhtml(bf(ARG1))\ 
            whenman(bf(ARG1))\ 
            whenms(bf(ARG1))\ 
            whentxt(bf(ARG1))\ 
            whensgml(bf(ARG1)))
    

  • The new macro is now in place. A new make job should install it all, and if necessary, make the appropriate new documentation files.

    After this, please kindly mail me your addition to the Yodl package. I will include it (if it's worth while), stating full credits too.




    Go back to index of Yodl.

    Please send Yodl questions and comments to yodl@icce.rug.nl.

    Please send comments on these web pages to (address unknown)

    Copyright (c) 1997, 1998, 1999 Karel Kubat and Jan Nieuwenhuizen.

    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.


    This page was built from Yodl-1.31.18 by

    <(address unknown)>, Wed Sep 2 06:14:50 2009 EDT.