zc zone compiler
================

## Overview ##

This is a small tool for generating DNS zones from relatively
simple text files, with some automation to handle complex or
repetitive tasks and to automate generation of reverse zone data.

`zc` ("zone compiler") is a Python script which uses an external
package (Bob Halley's excellent [dnspython][] toolkit) to do a lot of
the the heavy lifting.

`zc` can be used either as a straightforward command line tool or as a
pair of `pre-receive` and `post-receive` hooks in a bare git
repository.  In the latter mode, `zc` pulls its input data and
configuration directly from commits pushed to the master branch in the
git repository, using another external library ([GitPython][]).

Upshot of all this is that, once the git repository has been set up,
you just clone a copy of the repository, edit flat text files with
your favorite editor, commit and push, and you're done.  Compilation
will happen automatically when you push, any serious errors will abort
the push so you can fix them and try again, and output will be
installed automatically if there were no serious errors.


## Installation ##

`zc` is perfectly happy to run directly out of a clone of this source
code repository, but you can also install it using the usual tools:

### Python setuptools ###

You can install `zc` using the included `setup.py` in the usual
fashion:

```
python setup.py build
python setup.py install
```

The `setuptools` installation command takes a number of options
controlling things like whether you're installing to the base system
location, to a shared add-on location like `/usr/local/`, or to a
user-specific location.  Run `python setup.py install --help` for
details.

### Debian packaging ###

The source repository includes a basic Debian package setup for use on
Debian, Ubuntu, and related systems.  Building Debian packages is a
complex topic beyond the scope of this document, but if you have the
usual tools installed, something like this should work:

```
pdebuild --buildresult ..
debi --with-depends
```

You can of course also load the package into an APT repository,
install it directly with `dpkg -i`, whatever amuses you.

### Dependencies ###

As mentioned above, `zc` depends on `dnspython` and `GitPython`.  Both
of the packaging methods declare these dependencies in their
respective packaging structure, but you can also install the
dependencies directly if neceessary (eg, if running `zc` itself out of
its source tree).

setuptools:
```
pip install dnspython
pip install GitPython
```

Debian:
```
apt-get install python-dnspython python-git
```


## Command line use ##

If you just want to use `zc` as a command line tool, it's simple.
Usage as of this writing (subject to change, run `zc --help` for
current syntax):

    usage: zc [-h] [-o OUTPUT_DIRECTORY] [-l {debug,info,warning,error}]
	      input [input ...]

    Generate zone files from a simpl(er) flat text file.

    General intent here is to let users specify normal hosts in a simple
    and compact format, with a few utilities we provide to automate
    complex or repetitive stuff, including automatic generation of AAAA
    RRs based on a mapping scheme from A RRs.

    After generating the text of the forward zone, we run it through
    dnspython's zone parser, then generate reverse zones by translating
    the A and AAAA RRs in the forward zone into the corresponding PTR RRs.

    positional arguments:
      input                 input file

    optional arguments:
      -h, --help            show this help message and exit
      -o OUTPUT_DIRECTORY, --output-directory OUTPUT_DIRECTORY
			    directory for output files (default: .)
      -l {debug,info,warning,error}, --log-level {debug,info,warning,error}
			    how loudly to bark (default: warning)

You can supply more than one input file, `zc` will process them all
together before writing out any of the zone files.


## Use as git hooks ##

The following discussion assumes that you're keeping your `zc` input
files in a git repository.  We are *not* talking about the `zc` source
repository here: data files should be a *separate* bare git
repository.  You can put anything you like in that repository so long
as it obeys the rules below, but in most cases you will just want a
`config.json` file, one or more `zc` input files, and perhaps a README
explaining usage details and local conventions.

You'll want to create this repository on the server system where you
run the DNS name server which will be primary for the zone(s) you
administer with `zc`, so that it can install its output directly to
the name server.  If putting something like this on one of your public
name servers sounds scary, you can set `zc` up on a separate server
and run that as a stealth primary, with your public name servers as
secondaries (for the relevant zone(s)) of the stealth primary.

In most cases you'll want to set up a separate userid (`zc`, by
convention, but use whatever you like) to own the bare git repository,
so that you can set it up with a locked-down ssh configuration to
restrict git access to people authorized to change the zone.  The `zc`
package includes an auxiliary program `git-remote-only` suitable for
use in `~zc/.ssh/authorized_keys`.

To create a suitable userid and bare repository on Linux, you might do
something like this:

```
sudo adduser --disabled-password --gecos 'ZC user' zc
sudo git init --bare ~zc/dns.git
sudo mkdir ~zc/.ssh
sudo chown -R root:root ~zc
sudo chown -R zc:zc ~zc/dns.git
sudo ln -s /usr/bin/zc ~zc/dns.git/hooks/pre-receive
sudo ln -s /usr/bin/zc ~zc/dns.git/hooks/post-receive
```

You'll need to populate `~zc/.ssh/authorized_keys` to set up keys for
users who are allowed to use this repository.
`~zc/.ssh/authorized_keys` must be *readable* by user `zc`, but need
not be writable by user `zc`, so the ssh configuration can be owned by
root, and since it only contains public keys there's no particular
harm in making it world readable.  See comments in `git-remote-only`
for details on how to use that script.

When used as git hooks, configuration can't come from the command
line, so it comes from two places:

1. A file called `config.json` in the repository, and

2. Variables set in the configuration of the bare git repository where
   the `pre-receive` and `post-receive` hooks are installed.

The general idea is that stuff which should be under control of the
data owner is controlled by `config.json`, while stuff that should be
controlled by the server operator is controlled by configuration
variables in the server git repository.  These are described below.

We want to make sure that the zones compile correctly *before*
allowing the `git push` operation to complete, then, assuming
everything's OK, we want to install the zones *after* the commit
completes.

All the real work happens in the `pre-receive` hook, the
`post-receive` hook's job is just to trigger final installation of the
output zone files after `git-receive-pack` finishes accepting the
push.  If you don't understand git well enough to know what that
means, don't worry about it.  If you want to learn more, see
[githooks][], but the learning curve necessary for the documentation
to make any sense is a bit steep, so bring a bag lunch.

`zc` determines whether it's running as one of the git hooks or not by
examining the name by which it was invoked: if the name ends with
`/pre-receive` or `/post-receive`, it's a hook, otherwise you get the
command line behavior.  In practice, this means that you can just
install `zc`, symlink the correct hook names to whereever you
installed the `zc` script, and the right thing should happen.


### `config.json` settings ###

The list of input files and the verbosity are set in the JSON file,
while the output directory is set in the git configuration on the
server where the bare git repository lives.

    {
        "zones": ["foo.zone", "bar.zone"],
        "log-level": "info"
    }

The `zones` parameter is mandatory, and specifies the names of the
input files within the git repository (at the moment we only look at
the top-level directory -- we could change this given a reason).

The `log-level` parameter is optional, and defaults to `warning`.


### git server repository settings ###

All of the following settings have defaults except for `zc.output-directory`.

    git --git-dir /where/ever.git config zc.output-directory /my/output/directory

The `zc.output-directory` parameter in the git repository's
configuration file specifies the location of the directory to which
`zc` should write its final output.  `zc` also uses this directory to
stash a FIFO which it uses to coordinate actions between the
`pre-receive` and `post-receive` hooks.

There is no default for `zc.output-directory`, you must set it.

    git --git-dir /where/ever.git config zc.hook-timeout 15

`zc.hook-timeout` controls how many seconds the `pre-receive` hook
should wait for confirmation from the `post-receive` hook before
giving up.  The default value of 15 seconds should be fine unless your
server is really slow.

    git --git-dir /where/ever.git config zc.post-command 'rndc reload'

`zc.post-command`, if set, specifies a command to run after all
generated files have been installed.  The default is not to run any
such command.

    git --git-dir /where/ever.git config zc.log-file  /var/log/zc/zc.log
    git --git-dir /where/ever.git config zc.log-level warning
    git --git-dir /where/ever.git config zc.log-file-hours 24
    git --git-dir /where/ever.git config zc.log-file-count 7

When running in git hook mode, `zc` can log to both `stderr` (which
git passes back to the user executing the push) and to a log file.
The `zc.log-*` parameters control the log file.

`zc.log-file` is the name of the log file; if not set, `zc` will not
log to a file.

`zc.log-level` is optional, and defaults to `warning`.

`zc.log-file-hours` controls how many hours should elapse before `zc`
rotates its log file.  The default is 24 hours.

`zc.log-file-count` controls how many old log files `zc` should keep.
The default is 7.


## Zone generation ##

Other than `config.json`, the input files to `zc` look the same
regardless of whether you're running `zc` on the command line or via
git hooks.

While `zc` generates both forward and reverse zones, the underlying
mechanisms are (deliberately) very different, so it's simplest to
consider them separately.  Forward zones are driven from human-edited
files, while reverse zones are generated completely automatically from
the corresponding forward zones.


### Forward zone generation ###

Forward zone generation starts with a flat text file which is parsed
line-by-line to produce a forward zone file.  There three basic kinds
of lines in this file:

1. Stuff passed unchanged through `zc`: blank lines, comments, raw DNS
   RRs (for things other than addresses), and standard control
   operations like $TTL and $ORIGIN.

2. Name-address pairs, processed to generate A and AAAA RRs.

3. Control operations, all of which have names starting with "$".

`zc` requires that the text file start with a `$ORIGIN` control to
specify the name of the zone itself.

Other than the above, the one bit of processing `zc` performs is
replacement of the string "@SERIAL@ in an SOA RR with a
seconds-since-epoch integer timestamp.


#### Name-address pairs ####

"Name-address pairs" are exactly what they sound like: something that
a DNS zone file parser would consider a valid owner name, and and an
IP address (IPv4 or IPv6).

Processing of one name-address pair produces either one or two RRs,
depending on whether automatic generation of IPv6 addresses from IPv4
addresses is enabled when `zc` processes this name-address pair (see
the `$MAP` control operation, below).

Example:

    $MAP yes

    ; A couple of dual-stack hosts, with IPv6 addresses generated
    ; algorithmically from the IPv4 addresses.

    tweedledee          10.0.0.1
    tweedledum          10.0.0.2

    $MAP no

    ; Three single-stack hosts, addresses are what you see, RR type
    ; inferred from the address family

    larry               10.0.0.3
    moe                 2002:a00::4
    curly               10.0.0.5


#### `$MAP` and `$MAP_RULE` ####

The `$MAP` control operation enables or disables automatic generation
of IPv6 addresses from IPv4 addresses, according to zone-specific
mappings specified by the `$MAP_RULE` operation.

`$MAP` is simple: it takes one argument, `yes` or `no` (`on`,
`off`, `true`, and `false` are allowed as aliases).

`$MAP_RULE` takes two arguments: a prefix and a format string.  You
can specify `$MAP_RULE` more than once to build up an ordered set of
mapping rules.  When mapping is enabled, a given address will be
checked against the prefix of each rule in turn: the format string
from the first matching rule (if any) will be used to format the
mapped address.

Format strings are in the syntax used by Python's `str.format()`
operator; the `.format()` operator will be called with one argument,
the input address converted to a tuple of integers, one per byte in
the binary representation of the input address.  So an input address
of `10.0.0.44` would be yield the tuple `(10, 0, 0, 44)`, and so forth.

    $MAP_RULE   10.1.3.0/24             2002:a00:0000:f{0[2]}::{0[3]}
    $MAP_RULE   10.1.0.0/16             2002:a00:0000:{0[2]}::{0[3]}

    $MAP on

    larry       10.1.2.3
    moe         10.1.3.2

    $MAP off

    curly       10.1.4.1

This mechanism is intended primarily for mapping IPv4 addresses to
IPv6 addresses.  The mechanism itself is address-family-agnostic: in
principal, it should work equally well in the other direction if you
can specify a useful set of rules, but the author has not tested this.


#### `$RANGE` ####

The `$RANGE` control operation is a variation on the same general idea
as the (BIND9-specific) `$GENERATE` control operation, but is, in the
author's opinion, a bit easier both to use and to read.  For all but
the most esoteric uses, it takes three or four arguments:

1. A format string to generate the name field of the resulting RRs.

2. A start addresses (IPv4 or IPv6).

3. A stop address

4. An optional numeric `offset`.

The basic idea here is to generate a sequence of A or AAAA RRs (type
selected automatically to fit the addresses provided) for every
address in the specified range, inclusive, with names generated
according to a format string containing a numeric field.

An `offset` value of zero would start with the name generated by
applying the format string to the number zero; an `offset` value of
one would start with the name generated by applying the format string
to the number 1, and so forth.  If the `offset` field isn't specified
at all, it defaults to the numeric value of the least significant
octet of the start address.

Examples:

    ; Access points using $RANGE. This is equivalent to:
    ;
    ;  ap-101           10.0.1.101
    ;  ap-102           10.0.1.102
    ;  ...
    ;  ap-200           10.0.1.200

    $RANGE      ap-{:d} 10.0.1.101 10.0.1.200

    ; Switches, also using $RANGE, but with the numeric input to the
    ; format string explicitly specified rather than inferred from the
    ; IPv4 addressing.  Equivalent to:
    ;
    ; sw-1              10.0.3.17
    ; sw-2              10.0.3.18
    ; ...
    ; sw-26             10.0.3.42

    $RANGE      sw-{:d} 10.0.3.17 10.0.3.42 1

    ; Finally, a whole lot of DHCP client addresses, for IPv4
    ; addresses ranging from  10.1.0.50 to 10.2.255.254, names left as
    ; an exercise for the reader.

    $RANGE dhcp-f{:03x} 10.1.0.50 10.2.255.254 50


#### `$INCLUDE` ####

`$INCLUDE` is a standard control operator, but for the main expected
`zc` use cases there's not much need for it.

`zc` supports a limited form of the `$INCLUDE` operator, intended
mainly for automation (that is, for cases where one wants to include a
machine-generated set of DNS data into a larger zone that you're
maintaining with `zc`).  Limitations:

1. `zc` doesn't support the optional `origin` field of the `$INCLUDE`
   operator as defined in RFC 1035.

2. `zc` does *not* preserve the current `$ORIGIN` value of the outer
   file while processing `$INCLUDE`, so if the included file changes
   the `$ORIGIN`, the outer file will see that change.  Don't do that.

#### `$GENERATE` ####

The `$GENERATE` control operator is not currently implemented.

`$GENERATE` is a BIND-specific control operator.  We could implement
it if there were a real need, but the `$RANGE` operator covers the
things for which we have been using `$GENERATE` in the forward zone.

The IETF NOC's former use of `$GENERATE` in reverse zones was a source
of consistency problems, and is therefore unsupported with prejudice.


#### `$REVERSE_ZONE` ####

The `$REVERSE_ZONE` control operation has no effect on the forward
zone.  Rather, it's a mechanism for specifying the list of reverse
zones which should be generated from this forward zone.  We include
this in the input source for the forward zone in order to keep all the
data describing the zone in one place.

If you don't use the `$REVERSE_ZONE` control, `zc` will not generate
any reverse data for this forward zone.

Sample:

    $REVERSE_ZONE 0.10.in-addr.arpa
    $REVERSE_ZONE 1.10.in-addr.arpa
    $REVERSE_ZONE 2.10.in-addr.arpa
    $REVERSE_ZONE 0.0.0.0.0.0.a.0.2.0.0.2.ip6.arpa


### Reverse zone generation ###

As noted above, reverse zones are generated entirely from data
extracted from the forward zone.  This is deliberate: we are trying to
make sure that the reverse data corresponds to the forward data, and
giving the user an opportunity to get creative here is just asking for
trouble.

The basic strategy is:

* Create a reverse zone object for every name listed in the forward
  zone's source via `$REVERSE_ZONE` operators.

* For each `A` and `AAAA` RR in the forward zone, generate the
  corresponding `PTR` RR and and find the reverse zone in which that
  RR belongs; whine for each PTR RR that doesn't fit into any
  specified reverse zone.

* Populate the zone apex data (SOA and apex NS RRsets) of each reverse
  zone by copying the corresponding rdata from the forward zone.  Yes,
  this assumes that the forward and reverse zones are served by the
  same servers; we could "fix" that given a need, but as of this
  writing no such need exists, and this keeps it simple.


## Copyright ##

Copyright (c) 2017-2019, Grunchweather Associates

Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.


[dnspython]:    http://www.dnspython.org
[GitPython]:	https://github.com/gitpython-developers/GitPython
[githooks]:     https://git-scm.com/docs/githooks