Docco Next

Docco Next
§

Docco Next facilitates literate programming in several languages. It’s written in modern Javascript, and runs in Node.

See the generated documentation as HTML

Install and use
§

To use Docco Next run npm install -g docco-next and run it passing it a list of files (e.g. docco src/*js)

By default, every file will be converted into formatted HTML and will be placed in the default destination directory (docs by default). It’s also possible to convert files to Markdown, rather than HTML. Note that files will retain their original names and will change their extension to html.

Is it literate programming?
§

Formally speaking, literate programs put documentation strictly first.

In simple words, a literate file will look like a Markdown file, and its code is whichever Markdown code is included in the file. This means that anything that is not Markdown code (and is therefore indented in) is considered documentation, and programs are organised so that the documentation actually makes sense.

A notable example is CoffeeScript, which supports .litcoffee files natively.

However, not many languages support literate programming. Most languages will need a proprocessor, for example to convert the .js.md file into .js. lit is one of such tools. In Node, you can use Rich Harris’s lit-node to require .md files directly.

Docco Next works with normal source code (such as .js or .c) as well as literate source code (such as literate CoffeeScript, .litcoffee, or .js.md).

With pure literate style, all you have to do is add the .md extension to your source file.

When processing source files (such as .js or .c), one line comments are considered documentation and are parsed using Markdown. The code (that is, anything that is not a one-line comment) is processed by Shiki.

In both cases, the end result is a set of HTML pages with your documentation.

Similar projects
§

Here is a list of active projects which achieve similar goals:
- Docco by Jeremy Ashkenas (the same wise author of Underscore and Backbone). The program that inspired many others (including me) to use literate programming techniques and implement code that facilitates it
- pycco by Nick Fitzgerald. A Python implementation of literate programming
- Marginalia by Gary Deer. A Clojure implementation of literate programing. The project was originally started by Michael Fogus.
This file is the full source code of Docco Next, and it explains how it works in pure literate style.

Required files and starting configuration

The following libraries are required by Docco Next:

ejs. Used to convert the layout master file into HTML
globby. Used to glob for filenames when the shell doesn’t otherwise support this
fs-extra. Used for all I/O operations
marked. Used to convert Markdown into HTML
commander. Used to interpret command line parameters
shiki. Used to highlight source code

In Javascript terms, this becomes:

import path from 'node:path'
import { fileURLToPath, pathToFileURL } from 'node:url'
import ejs from 'ejs'
import { globby } from 'globby'
import fs from 'fs-extra'
import { marked, Marked } from 'marked'
import { gfmHeadingId } from 'marked-gfm-heading-id'
import { markedSmartypants } from 'marked-smartypants'
import { program } from 'commander'
import { createHighlighter } from 'shiki'
import { transformerNotationDiff } from '@shikijs/transformers'

Since ESM doesn’t provide __dirname and __filename by default, we recreate them using import.meta.url.

const __filename = fileURLToPath(import.meta.url)
const __dirname = path.dirname(__filename)

On startup, the version variable is worked out straight from the package.json file, which is loaded using fs-extra

const pkg = await fs.readJson(new URL('./package.json', import.meta.url))
const version = pkg.version

§

Docco Next manages several languages, stored in layouts/languages.json. Users can add more languages if desired. However, the list of default languages (and their relevant information) is stored in layouts/languages.json
```
const defaultLanguages = await fs.readJson(
  new URL('./layouts/languages.json', import.meta.url)
)
```
§

By default, marked is run with very basic options (basically, just turning smartypants on). Users can add more options, but this is the starting point
```
const defaultMarkedOptions = { smartypants: true }
```
§

Configuration

The run function is the entry point of the script when it’s run by the command line (that is, it’s not being used as a library). The docco command in bin simply runs import { run } from '../docco.js'; run().

The package commander will be used to parse command-line parameters. Commander will generate a configuration object based on how it’s configured using its option method.

For example the line:

.option('-L, --languages [file]', 'use a custom languages.json')

Means that if Docco Next is run with --languages ./some/file.json, the config object will include { languages: "./some/file.json"}.

Commander also provides the handy helpInformation() method, which will print out how the command is used.

This is the full set of options available in Docco Next:

async function run(args = process.argv) {
  program
    .name('docco')
    .version(version)
    .usage('[options] files')
    .option('-L, --languages [file]', 'use a custom languages.json')
    .option(
      '-l, --layout [name]',
      'choose a layout (default, parallel or classic)'
    )
    .option('-s, --shiki-theme [shikiTheme]', 'choose a light shiki theme')
    .option(
      '--shiki-theme-dark [shikiThemeDark]',
      'choose a dark shiki theme for dual-theme mode'
    )
    .option('--no-shiki-diff', 'disable shiki diff highlighting')
    .option('-o, --output [path]', 'output to a given folder')
    .option('-c, --css [file]', 'use a custom css file')
    .option('-p, --plugin [file]', 'use a custom plugin file')
    .option('-t, --template [file]', 'use a custom .ejs template')
    .option(
      '-e, --inputExtension [ext]',
      'assume a file extension for all inputs'
    )
    .option('-m, --marked [file]', 'use custom marked options')
    .option(
      '-x, --outputExtension [ext]',
      'set default file extension for all outputs'
    )
    .parse(args)

  const options = program.opts()
  if (program.args.length) {
    const config = { ...options, args: program.args }
    await cmdLineNormalise(config)
    await configure(config)
    await cmdLineSanityCheck(config)

    await documentAll(config)
  } else {
    return console.log(program.helpInformation())
  }
}

§

Three configuration functions are called:
- cmdLineNormalise() – expands some command line options to objects
- configure() – enriches configuration options with full paths etc.
- cmdLineSanityCheck() – checks that files and folders actually exist.
Here is an explanation of what each one does, followed by their source code.

Step 1: cmdLineNormalise()

It’s important to understand that Docco Next can be used as a library as well as a command line program. Two of the command line options, marked and languages, are external JSON files. When used as a library, Docco Next will expect marked and languages to be objects. However, when run as a command line program, those options will contain a string with a (JSON) file name instead.

The cmdLineNormalise() function is used to convert those strings into Javascript objects which will depend on the contents of the corresponding JSON files The fuction also assigns config.args (which is the list of files to be converted) to config.sources.

To sum up: when using Docco Next as a library, config.languages and config.maked will need to be objects. However, from the command line, they will be the paths of JSON files, and will be converted to objects by cmdLineNormalise(), which reads:

async function cmdLineNormalise(config) {
  if (config.languages) {
    if (!(await fileExists(config.languages))) {
      console.error('Languages file not found:', config.languages)
      process.exit(5)
    }
    config.languages = await fs.readJson(config.languages)
  }

  if (config.plugin) {
    if (!(await fileExists(config.plugin))) {
      console.error('Plugin file not found:', config.plugin)
      process.exit(5)
    }
    /**
     * Plugins are now loaded using dynamic `import()`.
     * We use `pathToFileURL` to ensure the path is correctly formatted
     * as a file URL, which is required for dynamic imports in ESM.
     */
    const pluginPath = path.isAbsolute(config.plugin)
      ? config.plugin
      : path.resolve(process.cwd(), config.plugin)
    const pluginModule = await import(pathToFileURL(pluginPath).href)
    config.plugin = pluginModule.default || pluginModule
  } else {
    config.plugin = {}
  }

  if (!config.outputExtension) config.outputExtension = 'html'

  if (config.marked) {
    if (!(await fileExists(config.marked))) {
      console.error('Marked file not found:', config.marked)
      process.exit(6)
    }
    config.marked = await fs.readJson(config.marked)
  }

  config.sources = await globby(config.args)
}

§

Step 2: configure()

This function is different to the other two cmdLineNormalise() and cmdLineSanityCheck(): rather than being a command line-only normalisation function, it’s actually a function that is run every time an API call is invoked. (Note that once it’s run once, it sets the config.configured property in the config object, and will check this property so that it will ever do anything only the first time it’s invoked)

This function is responsible to set sane defaults in the config property, so that each function doesn’t need to worry with wasteful checking.

For example config.output will be set as docs by default.

Also, for passed properties such as languages or marked, this function makes sure that the passed options are added to sane defaults. For example by passing the marked property in the config object, this function makes sure that the default smartypants: true is on, and any configuration options are added on top of what the user has passed. Or, that the config.languages contains the used-defined languages, as well as the default ones.

This function also sets some indirect configuration options that are not meant to be passed by the developer, but that are a result of the configuration.

Finally, this function also makes sure that the passed config makes sense: config.sources is scanned and any source with an unknown language (that is, a language not in the languages array) is filtered out (with a warning).

You can see this function as insurance that there is a solid, valid config object that every call can use. This is why every single function that uses config will call configure first.

async function configure(config) {
  if (config.configured) return
  config.configured = true

  config.output = config.output || 'docs'

  config.languages = properObjectWithKeys(config.languages)
    ? { ...defaultLanguages, ...config.languages }
    : defaultLanguages

  config.marked = properObjectWithKeys(config.marked)
    ? { ...defaultMarkedOptions, ...config.marked }
    : defaultMarkedOptions

  config.layout = config.layout || 'parallel'

  if (config.layout === 'parallel') {
    if (!config.shikiTheme) config.shikiTheme = 'github-light'
    if (!config.shikiThemeDark) config.shikiThemeDark = 'ayu-dark'
  } else if (!config.shikiTheme) {
    config.shikiTheme = 'github-light'
  }

  if (config.layout.match(/^[a-zA-Z0-9-]+$/)) {
    config.layout = path.join(__dirname, 'layouts', config.layout)
  }

  if (!config.css) config.css = path.join(config.layout, 'docco.css')

  config.public = path.join(config.layout, 'public')

  if (!config.template) {
    config.template = path.join(config.layout, 'docco.ejs')
  }

  config.sources = config.sources || []

  config.sources = config.sources.filter((source) => {
    const there = getLanguage(source, config)
    if (!there) {
      console.warn(
        `docco: file not processed, language not supported: (${path.basename(
          source
        )})`
      )
    }
    return there
  })
}

§

Step 3: cmdLineSanityCheck()

The cmdLineSanityCheck() is a command line-only function, which is used to make sure that paths specified in the config object have corresponding files or directories.

For example config.output represents the directory where all output files will be written. If that directory doesn’t exist, Docco Next (as run from the command line) will refuse to work. The same applies to config.layout (the path to the layout, which is effectively the “theme” used), config.css (the alternative CSS file used), and all files specified in the sources array.

async function cmdLineSanityCheck(config) {
  if (config.output && !(await dirExists(config.output))) {
    console.error('Output directory not found:', config.output)
    process.exit(1)
  }
  if (config.layout && !(await dirExists(config.layout))) {
    console.error('Layout directory not found:', config.layout)
    process.exit(2)
  }

  if (config.sources) {
    for (const source of config.sources) {
      if (source && !(await fileExists(source))) {
        console.error('source file not found:', source)
        process.exit(5)
      }
    }
  }
}

§

Going through each file via documentOne() and documentAll()

Up to this point, it’s all been about preparing the ground for the program to run: processing of command line options, sanitising the configuration object, and so on.

It’s time to get to the actual work: scanning the sources array, and actually get the documentation create and copied, which happens thanks to the calls documentAll() and documentOne().

First of all, some important file-related utility functions are created – some of them are very generic, while some others are very specific to Docco Next.

The functions are:

dirExists() – checks if a directory exists
fileExists() – checks if a file exists
finalPath() – works out the final path of a file, depeding of config.output and allowing a change of extension
copyAsset() – copies a file or a directory into config.output, preserving the path relative to the destination. So some/other/file.html will be copied to docs/some/other/file.html
write() – simply writes a file to its destination

There are also some utility functions that are not file-related:

properObjectWithKeys() – returns true if the passed variable is an proper (not-null) object that is not empty
getLanguage(source, config) - TODO: explain what it actually does

Here is the code for those functions:

async function dirExists(dir) {
  if (await fs.pathExists(dir)) {
    const stat = await fs.lstat(dir)
    return stat.isDirectory()
  }
  return false
}

async function fileExists(dir) {
  if (await fs.pathExists(dir)) {
    const stat = await fs.lstat(dir)
    return stat.isFile() || stat.isSymbolicLink()
  }
  return false
}

function finalPath(source, config) {
  const ext = config.outputExtension
  return path.join(
    config.output,
    path.dirname(source),
    path.basename(source, path.extname(source)) + '.' + ext
  )
}

async function copyAsset(file, type, config = {}) {
  await configure(config)
  if (!file) return
  if (type === 'file' && !(await fileExists(file))) return
  if (type === 'directory' && !(await dirExists(file))) return
  return fs.copy(file, path.join(config.output, path.basename(file)))
}

async function write(source, path, contents) {
  console.log(`docco: ${source} -> ${path}`)
  await fs.outputFile(path, contents)
}

function properObjectWithKeys(o) {
  return typeof o === 'object' && o !== null && Object.keys(o).length > 0
}

function getLanguage(source, config = {}) {

Note: configure is async now if we wanted, but here it might have been called already But let’s keep it simple.

  let codeExt, codeLang, lang

  const ext =
    config.inputExtension || path.extname(source) || path.basename(source)
  lang = config.languages[ext]
  if (!lang) return
  if (lang.name === 'markdown') {
    codeExt = path.extname(path.basename(source, ext))
    if (codeExt) {
      codeLang = config.languages[codeExt]
      if (codeLang) {
        lang = { ...codeLang, literate: true }
      }
    }
  }
  /* Add commentMatcher */
  lang.commentMatcher = new RegExp(`^\\s*${lang.symbol}\\s?`)
  /* Add commentFilter */
  /* Ignore [hashbangs](http://en.wikipedia.org/wiki/Shebang_%28Unix%29) and interpolations... */
  lang.commentFilter = /(^#![/]|^\s*#\{)/

  return lang
}

§

Now that everything really is ready, it’s time to (finally!) go through the source array and run documentOne() on each one of them. Also, if the sources are being converted to HTML, the css file and the public directory for that layout are copied over using the copyAsset utility function

Note that configure() is run. Since configure() only ever does anything the first time it’s run, and since it was already run in the run() function, it may seem superfluous to run it again. However, keep in mind that Docco Next exports an API. So, every call that receives config will always run configure(config) to make sure that sane defaults are set.
```
async function documentAll(config = {}) {
  await configure(config)
  await fs.mkdirs(config.output)

  for (const source of config.sources) {
    await documentOne(source, config)
  }

  await copyAsset(config.css, 'file', config)
  await copyAsset(config.public, 'directory', config)
}
```
§

documentOne is the centrepiece of the program: it uses the important parsing and manipulation functions (which are also available as API) to write the HTML file to the specified output path (docs by default) This function will:
- Read a file
- figures out which language it’s written in
- If a language is literate (see: .litcoffee or .md), convert it to code first using litToCode(). This is to ensure that parse() (the next call) always receives “normal”, runnable code (which is what it expects)
- Run parse() which will parse the file into an array of sections, where each element has the properties docsText and codeText
- Run formatAsHtml() which returns a formatted, finalised, ready-to-go HTML string based on the array of sections passed
- Write the file to its destination directory
Here is the source code:
```
async function documentOne(source, config = {}) {
  await configure(config)

  const buffer = await fs.readFile(source)
  let lines = buffer.toString().split('\n')
  const outputPath = finalPath(source, config)

  config.lang = getLanguage(source, config)
  if (!config.lang) {
    console.warn(
      `docco: file not processed, language not supported: (${path.basename(
        source
      )})`
    )
    return
  }
  if (config.lang.literate) {
    lines = litToCode(lines, config)
  }
  const sections = parse(source, lines, config)

  const result = await formatAsHtml(source, sections, config)

  await write(source, outputPath, result)
}
```
§

The actual parsing and manipulation
§

As explained above, documentOne() is Docco Next’s centrepiece, calling litToCode(), parse(), formatAsHtml, and finally write().

These functions are indeed the core functionalities of Docco Next.

litToCode()

This function takes literal code and returns “proper” source code that is ready to be executed. It does this by simply de-intending Markdown-indented code, and adding a comment marker to any other line. So, for example:

This is a literate .md file that contains code. This is the full
extent of the program:

    console.log("Hello world!")

This is it!

Is transformed into actually executable code:

# This is a literate .md file that contains code. This is the full
# extent of the program:

console.log("Hello world!")

# This is it!

This step will only be taken in cases where the input file is literate code, which is true for those languages in languages.json where the flag literate is set to true, or when the file is for example something.js.md – that is, a Markdown file that contains Javascript

function litToCode(lines, config) {
  const lang = config.lang
  const retLines = []
  const markdownIndented = /^([ ]{4}|[ ]{0,3}\t)/
  let inCode = lines[0] && markdownIndented.exec(lines[0])

  /** Remembering that a source code line are those without leading "   ":
   * add a comment marker at the beginning of each non-code line
   * Take out the leading "    " from every code line
   **/
  for (const [i, line] of lines.entries()) {
    /**
      Empty lines are a special case:
        - in code, they must stay blank.
        - In doc, they myst include the comment marker
    **/
    const emptyLine = /^\s*$/.test(line)
    /* Non-empty lines are added depending on them being code or doc. */
    if (!emptyLine) {
      inCode = markdownIndented.exec(line)
      retLines[i] = inCode
        ? line.slice(inCode[0].length)
        : lang.symbol + ' ' + line
      /* The concept of "empty" changes whether we are in code or not */
    } else {
      retLines[i] = inCode ? '' : lang.symbol
    }
  }
  return retLines
}

§

parse()

This function takes a string representing the source code, and returns an array of sections. Each section has two properties: docsText (the documentation text, formatted with Markdown) and codeText (the code displayed underneath the text in docsText).

Basically, each “section” is made up of two chunks: one formatted as Markdown, followed by a piece of code.

There are two main cases: one for non-empty lines, and one for empty lines.

Non-empty lines can be either “comments” (which will end up in docsText) or code (which will end up in codeText). It’s crucial to understand that the flow works by adding comment lines to docsText, then (once a code line is found) adding code lines to codeText, and when a comment line is found again a new section is created, the two variables are reset, and everythig starts again.

In more technical terms, the flow starts with empty docsText and codeText; the function will append comment lines to docsText (with codeText being empty) until a code line is encountered. At that point it will append to codeText; however, when the next comment line (destined to docsText) is encountered, since codeText is not empty, a new section containing docsText and codeText is created and both those variables are cleared; the commented line that triggered the new section is then added to docsText (with a now empty codeText), and everything starts again. Basically, in the flow you know if you are in the middle of code if codeText is empty.

Empty lines are… empty, both for doc and code. To know where to add the empty line (codeText or docsText), the same assumption as above is used: if there is anything in codeText, it means that the parser is in the middle of a code section, and therefore the empty line will be added to that code section.

function parse(source, lines, config = {}) {
  let codeText, docsText
  const sections = []

  const lang = config.lang
  docsText = codeText = ''
  let lineNumber = 0
  let startLineNumber = 1
  for (const line of lines) {
    lineNumber++
    /* If the line is not empty, it will either go in the code section */
    /* or the docs section, depending on whether the comment character was */
    /* found at the beginning of the line */
    if (line) {
      /* Case #1: it's a "comment" */
      /*  Text will go in docsText as documentation */
      if (line.match(lang.commentMatcher) && !line.match(lang.commentFilter)) {
        /*
          DETOUR: If there is code in codeText already, close off that section
          the section by pushing it into `sections` and zeroing
         `docsText` and `codeText`
        */
        if (codeText) {
          sections.push({ docsText, codeText, startLineNumber })
          docsText = codeText = ''
        }

        /* Add the line to the documentation (docsText) taking out the leading */
        /* comment marker */
        let cleanLine = line
        if (lang.symbol) {
          cleanLine = line.replace(lang.commentMatcher, '')
        }
        docsText += cleanLine + '\n'

        /* If the line was a new markdown section (`===`,  `---` or `##`), */
        /* close off that section */
        if (/^(---+|===+|#+.*)$/.test(cleanLine)) {
          sections.push({ docsText, codeText })
          docsText = codeText = ''
        }
        /* Case #2: it's not a comment */
        /* Note that from this moment on `codeText` is no longer empty, */
        /* which means that the next comment line (destined to docsText) will */
        /* trigger a new section */
      } else {
        if (codeText === '') {
          startLineNumber = lineNumber
        }
        codeText += line + '\n'
      }
      /* If it's an empty line, it will go either in the */
      /* code section or in the docs section. */
      /* We know we are in the code section by checking if */
      /* there is any code in codeText yet */
    } else {
      if (codeText) codeText += line + '\n'
      else docsText += line + '\n'
    }
  }
  sections.push({ docsText, codeText, startLineNumber })

  return sections
}

/**
 * ## codeToHtml()
 *
 * This function uses [Shiki](https://shiki.style/) to highlight source code.
 * It's an asynchronous function that returns the highlighted HTML.
 *
 * It utilizes several Shiki "transformers" to achieve the desired output:
 *
 * * `transformerNotationDiff()` -- parses `// [!code ++]` and `// [!code --]`
 *   comments to highlight added/removed lines. This can be disabled by
 *   setting `config.shikiDiff` to `false` (via the `--no-shiki-diff` CLI flag).
 * * `docco-render` -- a custom transformer that ensures the output matches
 *   Docco's specific layout requirements (background, line numbers, line IDs).
 *
 * The `config` parameter is used to check for the `shikiDiff` preference.
 */
async function codeToHtml(
  highlighter,
  code,
  language,
  lineNumber,
  config = {}
) {
  if (!code.trim()) {
    return ''
  }

  const transformers = []
  if (config.shikiDiff !== false) {
    transformers.push(transformerNotationDiff())
  }

  transformers.push({
    name: 'docco-render',
    pre(node) {
      node.properties.style = 'background-color: transparent'
    },
    code(node) {
      if (typeof lineNumber === 'number') {
        node.properties.style = `--line-start-number: ${lineNumber};`
      }
    },
    line(node, line) {
      if (typeof lineNumber === 'number') {
        node.properties.id = `L${lineNumber + line - 1}`
      }
      if (
        node.children.length === 0 ||
        (node.children.length === 1 &&
          node.children[0].type === 'text' &&
          node.children[0].value === '')
      ) {
        node.properties.class = (node.properties.class || '') + ' empty-line'
      }
    }
  })

  const themeOption = config.shikiThemeDark
    ? { themes: { light: config.shikiTheme, dark: config.shikiThemeDark } }
    : { theme: highlighter.getLoadedThemes()[0] }

  return await highlighter.codeToHtml(code, {
    lang: language,
    ...themeOption,
    transformers
  })
}

§

formatAsHtml()

This function takes the sections and format them as HTML, alternating docsText and (Markdown-indented) codeText

This function is essentially split into two subfunctions; they are both run passing on all of the parameters:

formatSections()
makeHtmlBlob()

The first one, formatSections(), will take the sections array; remember: each element has two properties, the leading docsText and the trailing codeText. After running formatSections(), each element will also have docsHtml and codeHtml (their respective HTML versions). This is done using markdown for the docs, and shiki for the code.

If a plugin was specified, the following functions from the plugin may be run:

plugin.beforeMarked – run before feeding the text to Marked. This allows users to extend Markdown as neeed.
plugin.afterHtml – run after the HTML has been generated
plugin.finalPass – run once the whole file has been generated and it’s ready to be written on the disk.

Since Markdown documentation can also contain code (by indenting 4 spaces), the shiki option is set for Markdown, instructing it what to do when a code block is encountered: obviously, the shiki library will be used to format it.

Since modern versions of Marked have moved many features to extensions, Docco Next manually enables the ones required to maintain its original functionality:

gfmHeadingId – restores automatic generation of ID attributes for headings
markedSmartypants – provides “smart” typographic punctuation (curly quotes, dashes, etc.)

The second functtion, makeHtmlBlob(), actually creates the final HTML code using the formatted sections as a starting point. The conversion is done by using the EJS template provided, and passing it important variables:

sources – it’s the list of sources, useful to create table of contents
css – it’s the path to the CSS file, relative to the processed file.
title – the title of the file. If the file starts with a markdown heading, this variable will have the contents of that heading; otherwise, it will have the file name.
firstSectionIsTitle – if true, the title variable is indeed the first section’s markdown heading. Templates can use this information to write the logic around the title.
sections – array of the various sections, which include docsHtml and codeHtml. This array is used by the EJS template to know what the file actually contains
finalPath and relativeToThisFile – two functions often used together in templates to know how to (HTML) link to another file in sources. For example relativeToThisFile(finalPath(source)).

One note about the _getTemplate() function. The aim of the function is to load the template file, and return an EJS compiler. The file itself is memoized, so that calling _getTemplate() doesn’t result in multiple reloading of the same file (since formatAsHtml() is potentially called multiple times, once for each passed file). Memoization avoids using a global variable.

async function formatAsHtml(source, sections, config = {}) {
  await configure(config)

  const lang = config.lang
  const highlighter = await createHighlighter({
    themes: config.shikiThemeDark
      ? [config.shikiTheme, config.shikiThemeDark]
      : [config.shikiTheme],
    langs: [
      'javascript',
      'typescript',
      'python',
      'ruby',
      'c',
      'cpp',
      'java',
      'go',
      'rust',
      'markdown',
      'json',
      'css',
      'html'
    ] // Add more as needed, or dynamic
  })

  /* [Markdown](https://github.com/markedjs/marked) and Shiki */
  /* In modern marked, we use extensions for highlighting */
  const localMarked = new Marked()

  /* We use the `gfmHeadingId` extension to restore automatic generation */
  /* of ID attributes for headings, which is no longer part of core Marked */
  localMarked.use(gfmHeadingId())

  /* If the `smartypants` option is set (which it is by default), we use */
  /* the `markedSmartypants` extension to provide "smart" typographic punctuation */
  if (config.marked?.smartypants) {
    localMarked.use(markedSmartypants())
  }

  /* Custom renderer and async token walker for code blocks in markdown */
  /* Code might happen within the markdown documentation as well! If that */
  /* is the case, it will highlight code either using the language specified */
  /* within the Markdown codeblock, or the default language used for the processes */
  /* file. */
  /**
   * Since Shiki's highlighting is asynchronous, we use Marked's `walkTokens`
   * and `async: true` mode. This ensures all code blocks are highlighted
   * before the final HTML rendering starts, preventing `[object Promise]`
   * from appearing in the output.
   */
  localMarked.use({
    async walkTokens(token) {
      if (token.type === 'code') {
        const displayLang = token.lang || lang.name
        try {
          token.renderedCode = await codeToHtml(
            highlighter,
            token.text,
            displayLang,
            undefined,
            config
          )
        } catch {
          console.warn(
            `${source}: language '${displayLang}' not recognised, code block not highlighted`
          )
          token.renderedCode = `<pre><code>${token.text}</code></pre>`
        }
      }
    },
    renderer: {
      code(token) {
        /* The renderer returns the pre-rendered HTML stored on the token. */
        return token.renderedCode
      }
    },
    async: true
  })

  if (config.marked) {
    localMarked.use(config.marked)
  }

  /* Format and highlight the various section of the code */
  for (const section of sections) {
    let code = ''
    try {
      code = await codeToHtml(
        highlighter,
        section.codeText,
        lang.name,
        section.startLineNumber,
        config
      )
    } catch {
      code = `<pre><code>${section.codeText}</code></pre>`
    }

    code = code.replace(/\s+$/, '')
    if (code !== '') section.codeHtml = `${code}`
    else section.codeHtml = ''

    let docsText = section.docsText
    if (config.plugin.beforeMarked) {
      const newText = await config.plugin.beforeMarked(docsText)
      docsText = newText
    }

    section.docsHtml = await localMarked.parse(docsText)

    if (config.plugin.afterHtml) {
      const newHtml = await config.plugin.afterHtml(section.docsHtml)
      section.docsHtml = newHtml
    }
  }

  /* return the HTML blob */
  return makeHtmlBlob(source, sections, config, lang)
}

/* Once all of the code has finished highlighting, we can **write** the resulting */
/* documentation file by passing the completed HTML sections into the template, */
/* and rendering it to the specified output path. */
async function makeHtmlBlob(source, sections, config, lang) {
  let first

  async function _getTemplate(templatePath) {
    if (formatAsHtml._template) return formatAsHtml._template
    const templateContent = await fs.readFile(templatePath, 'utf8')
    formatAsHtml._template = ejs.compile(templateContent)
    return formatAsHtml._template
  }

  const thisFile = finalPath(source, config)

  function relativeToThisFile(file) {
    const from = path.resolve(path.dirname(thisFile))
    const to = path.resolve(path.dirname(file))
    return path.join(path.relative(from, to), path.basename(file))
  }

  function includeText(s, silentFail) {
    try {
      return fs.readFileSync(s, 'utf8')
    } catch (e) {
      if (silentFail && e.code === 'ENOENT') return ''
      if (e.code === 'ENOENT') {
        console.error('Could not load included file:', s)
      } else {
        console.log(e)
      }
      process.exit(100)
    }
  }

  /* Work out `title`, which will be either the first heading in the */
  /* documentation, or (as a last resort) the file name */
  const firstSection = sections.find((s) => s.docsText.length > 0)
  let lexed
  if (firstSection) {
    lexed = marked.lexer(firstSection.docsText)
    first = lexed[0]
  }
  const maybeTitle = first && first.type === 'heading' && first.depth === 1
  const title = maybeTitle ? first.text : path.basename(source)
  const firstSectionIsTitle = maybeTitle && lexed.length === 1

  /* If the first section is the title, then get rid of it  */
  /* since the title is already being displayed by the template anyway */
  if (firstSectionIsTitle) {
    sections.shift()
  }

  /* The `css` variable will be available in the template as a relative */
  /* link to the CSS file */
  const css = relativeToThisFile(
    path.join(config.output, path.basename(config.css))
  )

  const template = await _getTemplate(config.template)

  /* Make up the HTML based on the template */
  let html = template({
    lang,
    includeText,
    source,
    sources: config.sources,
    css,
    firstSectionIsTitle,
    title,
    sections,
    finalPath: (p) => finalPath(p, config),
    relativeToThisFile,
    hasTitle: firstSectionIsTitle,
    destination: (p) => finalPath(p, config),
    relative: relativeToThisFile
  })

  /* Run the final pass */
  if (config.plugin.finalPass) {
    html = await config.plugin.finalPass(html)
  }

  return html
}

§

Public API
§

These functions are available once the module is required(). They are self contained and can be used to take Docco Next to different directions.
```
export {
  copyAsset,
  run,
  parse,
  formatAsHtml,
  litToCode,
  documentOne,
  documentAll,
  version
}
```

Docco Next

Install and use

Is it literate programming?

Similar projects

Required files and starting configuration

Configuration

Step 1: `cmdLineNormalise()`

Step 2: `configure()`

Step 3: `cmdLineSanityCheck()`

Going through each file via documentOne() and documentAll()

The actual parsing and manipulation

litToCode()

parse()

formatAsHtml()

Public API

Docco Next

Install and use

Is it literate programming?

Similar projects

Required files and starting configuration

Configuration

Step 1: cmdLineNormalise()

Step 2: configure()

Step 3: cmdLineSanityCheck()

Going through each file via documentOne() and documentAll()

The actual parsing and manipulation

litToCode()

parse()

formatAsHtml()

Public API

Step 1: `cmdLineNormalise()`

Step 2: `configure()`

Step 3: `cmdLineSanityCheck()`