User Guide - Data Backends

  • Overview
  • For Developers
  • For Authors
  • Documentation
  • Download
  • Blueprints
  • Data Backends

    Backends are the things that pull data into your website. Such data can come from diverse sources, and in various content formats. Sprinkles makes a best effort at normalizing things into a consistent model, such that your templates don’t need to know whether the data they’re handling was originally a local .docx file, a JSON document fetched from a RESTful API, or a result set from an SQL query.

    There are two forms for backend definitions: long-hand (objects) and short-hand (strings). The string form is more convenient, but doesn’t offer all the options.

    Supported Backend Types

    All backend specifications support at least the following keys in long-hand mode:

    The file Backend

    Fetches data from local files.


    type: 'file' / 'glob' / 'dir'
    path: '{filename}' # The filename, absolute or relative, to load


    'file://{filename}' / 'glob://{glob-expr}' / 'dir://{dirname}'

    MIME type detection: by file extension.

    This backend comes in three flavors:


    type: file
    path: '/var/www/example.org/static/{staticfile}'

    The http Backend

    Issues HTTP GET request to a remote server.


    type: 'http' / `https`
    uri: '{remote-uri}'


    'http://{remote-uri}' # remote URI (without the `http://` prefix)

    Using https instead of http will fetch data over an HTTPS connection.

    MIME type detection: from response header.

    Default: fetch: one

    The sql Backend

    Fetches data from an SQL database. Currently supports SQLite3 and PostgreSQL.


      type: 'sql'
        driver: {driver} # one of 'sqlite', 'postgres'
        dsn: {driver-specific data source name}
      query: {SQL query} # parametrized using ? for placeholders
      params: [ "{{param1}}", "{{param2}}", ... ] # parameters to put in placeholder



    Note that for security reasons, the SQL query string itself does not support variable substitution; captured variables can only be interpolated into the params array. This is to prevent SQL Injection: captured variables are user-supplied and thus potentially tainted, so we only accept them into parameters, not into the raw query. Since the shorthand form doesn’t cater for parameters, it follows that the shorthand doesn’t support injecting captured variables.

    Because SQL result sets do not come in any particular file format by themselves, sprinkles pretends they’re JSON, and exposes them with a content type of text/json, as a document containing a list of rows, each modelled as an object of column names to values.

    MIME type detection: hard-coded as application/json

    Default: fetch: all

    The subprocess Backend

    Run a program locally, and return its output (read from stdout).


    type: 'subprocess'
    cmd: ['{program}', '{arg1}', '{arg2}', ... ]
    mime-type: '{mime-type}'

    Shorthand: n/a

    Particular points of interest:

    MIME type detection: manual (from mime-type parameter)

    Default: fetch: one

    The post Backend

    Parse the request body according to its content type.


    type: 'post'



    Particular points of interests:

    MIME type detection: from request header.

    Default: fetch: one

    The literal Backend

    Return inline data literally.


    type: 'literal'
    value: {any valid YAML value}


    'literal://{string value}'

    Note that the shorthand form only supports string values, but the longhand can take any valid YAML data structure, and will pass it into the template as-is.

    MIME type detection: from data (JSON strings: text/plain;charset=utf8, anything else: application/json).

    Variable Backends

    Both definition forms (short-hand and long-hand) support interpolating captured variables in order to parametrize backend queries (i.e., fetch different data items based on the variable parts of the matched route). In fact, the interpolation uses Ginger, the same language that you’ll use for the page templates.

    Variables can come from captured route elements (e.g. if your route is /pages/{{*:page_name}}, then writing {{page}} anywhere in a backend definition will inject the page name captured from the request path); they can also come from other backends, as long as they load before the current one.

    CAVEAT 1

    In order to force loading order, backend definitions can be written in a two-stage data structure, namely a list of (ideally single-element) objects, like so:

      - meta: 'file://data/meta/{{page}}.yml'
      - content: 'file://data/content/{{meta.filename}}'

    Written in JSON syntax, the structure is a bit clearer:

    "data": [
      { "meta": "file://data/meta/{{page}}.yml" },
      { "content": "file://data/content/{{meta.filename}}" }

    This is because plain YAML objects are conceptually unordered, so the order in which they are written in the file does not survive the YAML parsing step. But by putting them in a list, we can control the order in which they are processed.

    Without this trick, the content backend might end up being processed before the meta backend, and that would mean that interpolating meta.filename into the definition for content could not possibly work.

    CAVEAT 2

    Due to the nature of Ginger templates, the literal backend will not retain binary data associated with values produced in templates. For example, the lilypond filter is capable of producing raw binary PNG data, but if you do something like this:

      - file:
          type: literal
          body: "{{music|lilypond(raw=true)}}"
          force-mime-type: "image/png"

    …then that won’t work, because the output from the lilypond filter is converted to plain text (not binary!) in order to inject it into the body, and for something like PNG data, that simply cannot work.

    Supported Content Types

    Sprinkles detects the content type (file format) for backend data automatically in most cases, depending on the backend type (see above). All the supported MIME types are marshalled into the same format, but due to the diverse nature of the data, details may still differ. The following types are currently supported:

    Getting Embedded Media Out Of Complex Documents

    Some Pandoc document types (currently .docx) contain not only a rich-text body, but also embed media such as images, which are referenced from within the document itself. When rendering the body, we faithfully reproduce these references, typically as <img> tags, but in order to actually get those images into the resulting web page, we need a bit of extra work in the form of a special routing rule.

    For a document served something like this:

    - pattern: '/{{page:*}}'
        - page: 'file://./data/pages/{{page}}.*'
      template: 'page.html'

    …we need an additional routing rule that makes it such that relative links in <img> tags end up serving the images from the document statically. For this to work, we need two things: a suitable routing rule, such that appending the filename of the embedded image to the parent document’s URL matches, and an extra property on the rule to tell the static file handler that we want to serve a child item rather than the parent document. This is what such a rule looks like:

    - pattern: '/{{page:*}}/{{mediapath:**}}'
        - file: 'file://./data/pages/{{page}}.*'
      static: true
      child: '{{mediapath}}'