Jump to Content

Highlighting PHP Source Code

Often it is necessary to present PHP source code via XHTML, for documentation or (as in these articles) to illustrate a point. When presenting such code, we require:

  1. Valid XHTML
  2. Semantic Validity (<code> tags)
  3. Syntax highlighting
  4. Line numbering
  5. Easy for viewers to copy and paste code

So, what are our options? PHP provides an inbuilt function to highlight and display code, highlight_string. However, the XHTML output from this is, frankly, a mess. Although it produces valid XHTML with highlighted syntax, there is no line numbering, and styling is embedded into the source code. It is possible to use this function and then clean up the output — the best example of this approach is probably Chris Shiflett's highlighting code. He encapsulates the code in an XHTML ordered list, and uses spans to markup the syntax:

XHTML

  1. <ol>
  2. <li><code> ...
  3. <span class="comment"> ... </span>
  4. ... </code></li>
  5. <li><code> ... </code></li>
  6. </ol>

The advantage of this approach is that it creates line numbers automatically, and still allows viewers to copy and paste easily from the presented code. As valid semantic XHTML, it therefore hits all our targets, and is a good solution to the problem of highlighting source code. However, while I think the markup is excellent, the methodology bugs me a little: re-parsing the original output from highlight_string is a hack, and makes the resulting code fragile. The output from highlight_string is already different between PHP4 to PHP5, what happens if it is changed again? There must be a better way. There is, and indeed Chris mentions it in his original article: the PHP tokenizer library.

The rest of this article explains how this library can be used to produce highlighted source code as XHTML. There are a number of niggly problems which need to be overcome, and the impatient among you may want to skip ahead directly to our final PHP highlighting using the tokenizer library example.

Laying the Foundations

The PHP tokenizer library provides access to the Zend lexical language parser, and is enabled by default. The flagship function token_get_all converts a source code string into an array of component pieces like variables, strings, keywords, and so on. This array can then be parsed to produce a markup similar to Chris' original.

The first step is to write a function that will convert the 'tokens' into a classname. There are quite a number of different tokens, which gives us very fine-grained control over the syntax highlighting, but makes for a rather verbose function. If we want to markup comment, variable, string, number, tag and content outside PHP tags separately, this function might be:

PHP 5

  1. <?php
  2. function getClassFromToken($type)
  3. {
  4. switch ($type) {
  5. case T_OPEN_TAG:
  6. case T_OPEN_TAG_WITH_ECHO:
  7. case T_CLOSE_TAG:
  8. return 'tag';
  9. case T_COMMENT:
  10. case T_DOC_COMMENT:
  11. return 'comment';
  12. case T_INLINE_HTML:
  13. return 'inline-content';
  14. case T_CONSTANT_ENCAPSED_STRING:
  15. return 'string';
  16. case T_VARIABLE:
  17. case T_STRING_VARNAME:
  18. return 'variable';
  19. case T_LNUMBER:
  20. case T_DNUMBER:
  21. return 'number';
  22. case T_ABSTRACT:
  23. case T_ARRAY:
  24. case T_AS:
  25. /* ...
  26. a number of keyword tokens
  27. have been omitted for brevity!
  28. ...
  29. */
  30. case T_TRY:
  31. case T_UNSET:
  32. case T_WHILE:
  33. return 'keyword';
  34. default:
  35. return false;
  36. }
  37. }
  38. ?>

The source code is parsed into tokens by the function token_get_all that returns an array of source pieces. If the source piece has a token, an array of content and token value is returned. If no token can be associated with that piece, the plain string is returned. For our purposes, we need to normalise the value for each piece to get the content and class name.

PHP 5

  1. <?php
  2. $tokens = token_get_all($source);
  3. foreach ($tokens as $piece) {
  5. /* standardise token */
  6. if (is_array($piece)) {
  7. list($type,$content) = $piece;
  8. $class = getClassFromToken($type);
  9. } else {
  10. $type = false;
  11. $class = false;
  12. $content = $piece;
  13. }
  15. /* $type == token type
  16. $class == class name required
  17. $content == source string */
  19. }
  20. ?>

At this point we are ready to tackle the main task of iterating over the tokenized source and encapsulating it in an ordered list, highlighting syntax with XHTML class spans. Line breaks in the original source are simply treated as whitespace by the tokenizer library and need to be explicitally detected by the iteration code.

PHP 5

  1. <?php
  2. $tokens = token_get_all($source);
  3. $out = '<ol class="php">'.PHP_EOL.'<li><code>';
  4. foreach ($tokens as $piece) {
  6. /* standardise token:
  7. $type == token type
  8. $class == class name required
  9. $content == source string */
  11. /* act on token. */
  12. $content = explode("\n",$content);
  13. for ($i=0,$max=count($content)-1; $i<=$max; $i++) {
  15. /* add new line on 2nd iteration or greater */
  16. if ($i>0) {
  17. $out .= '</code></li>'.PHP_EOL.
  18. '<li><code>';
  19. }
  21. /* trim if at end of line */
  22. if ($i<$max) {
  23. $content[$i] = rtrim($content[$i]);
  24. }
  26. /* wrap content in spans */
  27. if ($class !== false) {
  28. $out .= '<span class="'.$class.'">'.
  29. htmlentities($content[$i]).
  30. '</span>';
  31. } else {
  32. $out .= htmlentities($content[$i]);
  33. }
  35. }
  36. }
  37. $out .= '</code></li>'.PHP_EOL.'</ol>';

We have now written the core of the source highlighting process, but there is still one situation where the current code will fail, and that is within a doubled-quoted string which has embedded variables.

Dealing with Variables within a Double-Quoted String

There are two token types associated with strings, T_STRING and T_CONSTANT_ENCAPSED_STRING. The latter token is associated with single quoted strings, or double quoted strings with no embedded variables. The T_STRING type is associated with all plain strings located within the source, strings such as function names, class names, and so on. We want to highlight only quoted strings, thus only the T_CONSTANT_ENCAPSED_STRING type returns a 'string' classname from our getClassFromToken function. However, when a double-quoted string contains embedded variables, the quoted string is broken down into a number of sub-tokens:

PHP 5

  1. <?php
  2. $source = '<?php "embedded $variable"; ?>';
  3. print_r(token_get_all($source));
  4. /*
  5. Array (
  6. [0] => Array( [0] => T_OPEN_TAG
  7. [1] => <?php )
  8. [1] => "
  9. [2] => Array( [0] => T_STRING
  10. [1] => embedded )
  11. [3] => Array( [0] => T_WHITESPACE
  12. [1] => )
  13. [4] => Array( [0] => T_VARIABLE
  14. [1] => $variable )
  15. [5] => "
  16. [6] => ;
  17. [7] => Array( [0] => T_WHITESPACE
  18. [1] => )
  19. [8] => Array( [0] => T_CLOSE_TAG
  20. [1] => ?> )
  21. )
  22. */
  23. ?>

So when a double quoted string is broken into sub-pieces we want to highlight T_STRING tokens within the double quotes. An active double quote is always parsed into a piece by itself, so we simply need to track whether a double quote is open or closed and treat T_STRING appropriately:

PHP 5

  1. <?php
  2. $tokens = token_get_all($source);
  3. $in_quotes = false;
  4. foreach ($tokens as $piece) {
  6. /* standardise token
  7. $type == token type
  8. $class == class name required
  9. $content == source string */
  11. /* account for embedded variables */
  12. if ($type === false && $content === '"') {
  13. $class = getClassFromToken(
  14. T_CONSTANT_ENCAPSED_STRING);
  15. $in_quotes = !$in_quotes;
  16. } elseif ($in_quotes && $type === T_STRING) {
  17. $class = getClassFromToken(
  18. T_CONSTANT_ENCAPSED_STRING);
  19. }
  21. /* parse token piece to XHTML */
  23. }
  24. ?>

And voila! All quoted strings are highlighted appropriately, irrespective of whether they have embedded variables or not:

PHP 5

  1. <?php
  2. echo "embedded $variable in string";
  3. echo "embedded {$array['key']} in string";
  4. ?>

The Final Highlighter Script

Bring together all the elements discussed so far into a single class, add in some odd-even line highlighting, sprinkle in some CSS magic... And we arrive at the final PHP highlighting using the tokenizer library example. This validates to XHTML Strict Standards, numbers the source lines, highlights PHP syntax and allows viewers to copy and paste the code easily.

Snapshot

How to highlight PHP source code listings in semantic XHTML. When generating source code listings for documentation or articles, the source needs to be converted to highlighted and formatted XHTML. This article shows how this can be accomplished using the PHP tokenizer functions.

Related Links

Share

Articles

Who we are

Openknot is the web design agency of Rob Tuley. We specialise in creating visually clean, dynamic sites with PHP and MySQL.

Find out more

Find out what we do. Browse our previous work. Take a closer look at who we are. Enjoy our articles. We hope you'll see we are passionate about our work.

Get in touch

Want us to work for you? Have a question about what we do?