$Id$ -*- Text -*-

Python RPKI production tools.

Requires Python 2.5.

External Python packages required:

- lxml, which in turn requires the libxml2 C libraries.

  FreeBSD: /usr/ports/devel/py-lxml

- MySQLdb, which in turn requires MySQL client and server.  I'm
  testing with MySQL 5.1.

  FreeBSD: /usr/ports/databases/py-MySQLdb

- TLSLite, which pulls in other crypto packages.

  FreeBSD: /usr/ports/security/py-tlslite

- Cryptlib, at the moment just to support TLSlite but may end up using
  it for other things later.

  FreeBSD: /usr/ports/security/cryptlib

  ...but the FreeBSD port doesn't (yet?) install the Python bindings,
  sigh, so at the moment you have to do that by hand:

  # cd /usr/ports/security/cryptlib
  # make install
  # cd work/bindings
  # python setup.py install
  # cd ../..
  # make clean

- Eventually I expect that this will require an event-handling package
  like Twisted, but I'm not there yet.

- The testpoke tool (up-down protocol command line test client) also
  uses PyYAML, mostly for compatability with APNIC's equivalent tool.

  FreeBSD: /usr/ports/devel/py-yaml

We also use a hacked copy of the Python OpenSSL Wrappers (POW)
package, but our copy has enough modifications that it's expanded in
the Subversion tree.  Depending on how this all works out, I may end
up splitting the POW.pkix module out of the POW package and using it
with Cryptlib, as the POW.pkix package is 98% about doing ASN.1 in
pure Python and only 2% about any kind of crypto.



Current TO DO list:

- Representation of timestamps is a mess.  We have four different
  kinds already: seconds from epoch, the the two flavors of timestamps
  used in ASN.1, and the timestamps used in MySQL.  Need a unifying
  class to hide all this nastiness.

  POW.pkix provides conversion functions for ASN.1.   MySQL timestamps
  appear to map to Python datetime objects.  One can convert between
  datetime and time (seconds since epoch) objects using:

    time_object = time.mktime(datetime_object.timetuple())
    datetime_object = datetime.datetime.fromtimestamp(int(time_object))

  if one is willing to discard fractional seconds (which we probably
  are in this case, as they're not useful with ASN.1).  No doubt there
  would be a way to preserserve fractional seconds if we cared.

  Except that the above still requires the time module's idiocy of
  setting the TZ environment variable to avoid having everything
  whacked to local time.  The time.time() call itself is fine, but
  most of the rest of the time module is for the birds.   If we're
  going to consolidate by subtyping datetime.datetime, we can avoid
  the conversion idiocy in the time module entirely by doing:

    time_object = int(datetime_object.strftime("%s"))

  which is probably even faster, albiet sillier.

- Whack expiration dates of certs to match irdb valid_until value when
  issuing -- valid_until is optional, what do we do if it's not set?
  Default period in self object seems obvious answer, neither Randy
  nor I has thought of anything better yet.

- Subsetting (req_* attributes in up-down protocol)

- Revocation and CRL generation

  - Need to keep data on unexpired revoked certs to generate CRL
    (added a timestamp for this, sufficient?)

  - Do we ever need to delay revocation of old certs to give their
    replacements time to propegate?

  These two imply that we need fields in child_cert table to
  indicate whether a cert is dead, eg, a date field which is NULL if
  the cert is still live, otherwise is the date after which it should
  be added to the CRL.

  Yes, but... every example of delayed revocation in the Common
  Management Tasks page in the APNIC Wiki is triggered by the child,
  not the parent.  This suggests that revocation by the parent is
  always immediate, whether it's due to overclaim or a child's class
  going away or a child requesting revocation (as in key rollover).

  So apparently it's the child (probably ca_detail object) that needs
  timestamps.  Other text in Common Management Tasks suggests that
  ca_detail also needs deferred transitions from pending to active.

  How to model this?   An enumerated state value (already have that,
  may need to rename states) and a timestamp (which may be NULL) for
  next scheduled transition?   Is the next state always predictable
  from the current state or do we also need a next-state field?

    state     := pending | active | deprecated | revoked
    timestamp := null | <time of next transition>

- Publication protocol and implementation thereof.   Defer until core
  functionality in the main engine is done.

  As an interim measure, hack some kind of stub publication (not real
  protocol yet, just dump to local filesystem so can see outputs and
  maybe rcynic against them); this is a stop-gap to let me concentrate
  on the main engine and defer work on the publication protocol and
  engine.

- Publication hooks everywhere - need not wait for protocol, can just
  log what would happen for now, or write to local file store (perhaps
  even in a form that we can use with rcynic as a relying party).
  Hooks for this go into:

  - Cert publication

  - CRL publication

  - Manifest publication

  - Withdrawal of any of the above

- Child batch processing loop, eg, regeneration or removal of expired
  certs, CRL update, manifest update, etc.  This should probably be an
  iteration over CA objects, as the CA is the actor in pretty much
  everything that might need to be done.

  Figuring out whether to regenerate or remove expired certs requires
  some of the same data as CRL generation.

  - Code to clean up expired certs

  - Code to revoke certs -- need to sort out when we do this
    automatically vs waiting for explicit revoke PDU from child

  - Code to generate CRLs

- Haven't done anything about db.commit() and db.rollback() yet, for
  that matter haven't yet whacked MySQL to enable those features.

- In theory, all access to object data attributes ought to be through
  accessor methods so that the .set() method can mark teh object as
  SQL-dirty automagically.  Not done yet.  One way of hiding the
  grotty bits here might be to make attributes look like elements of a
  dict(), ie, implement __getitem__() and __setitem__() methods,
  probably along with the other "container type" special methods for
  completeness.  An even easier way to do this would be to subclass
  dict() and just customize __setitem__().

  But all of this assumes that we really want automagic SQL-dirty, and
  I'm less convinced of that now than when I made the above note.

- Test with larger data set -- Tim gave me plenty of data and I have
  the low-level tools, just haven't written the glue logic to create
  child objects for all the entities in the IRDB, poll on behalf of
  each of them, and check the result for sanity

- Need to figure out how we're going to handle "root" case (IANA/RIR
  self-signed resource cert issuing to its children).  Right now this
  is a separate little program, which is nice for testing but feels
  wrong.  As Randy puts it, this is kind of a special case of a
  parent, although we might want to represent it differently.  If this
  were really only IANA and the RIRs, handling it as a special case
  separate program might make sense, but anybody who wants to certify
  private address space is going to have this problem, and the
  separate daemon approach just feels wrong for that.

  If it's not a separate daemon, will need left-right protocol support
  to configure whatever it is we're going to configure, with all the
  usual private key hygiene issues.  This probably implies a level of
  indirection, eg, the self-signed cert is generated in the IRBE, the
  RPKI engine generates PKCS#10 for a working cert to be issued by the
  self-signed cert (perhaps with RFC 3779 inheritance for everything,
  to keep it small), so that the RPKI engine never needs to hold the
  private key for the root.

  Deferred for the moment, not sure for how long.

- Logging subsystem, including syslog support.

Once this lot is done we'll be close to something that shows at least
the basics of normal operation, albiet in a form that's not yet usable
in production.

Follow-up after that will be getting rid of remaining synchronous code
(make daemon fully event-driven, except perhaps for SQL queries),
address rollback, commit, and other data integrity issues, and see how
well the resulting code handles hosting (multiple self objects in same
daemon).

Somewhere along the way I'll need to update to the new model of trust
anchors we ended up with in Amsterdam, first step for which will
involve writing it down (well, RobK was supposed to do that, but I was
supposed to convert some pencil sketches into graphviz for him so
we're both lame on this so far).  I don't think this results in major
changes, probably a few extra cert fields in the self object which we
then need to toss into the rpki.x509.X509_chain objects before
verifying CMS or TLS, and perhaps the existing TA fields in various
objects become pairs of certs instead of a single TA, but this is
mostly just generalization and reuse of existing code, no bold new
adventures.

At some point we need to do performance testing.  All of our crypto is
done internally except for CMS, for which we still call the OpenSSL
CLI tool.  The simplest solution would probably be to extend POW to
support CMS; cryptlib might be more entertaining solution, but am not
sure it's worth the time sink.

Although it may not be worth the effort, it'd be nice to get back to
only one crypto library.  As far as I know the only things we're using
M2Crypto or cryptlib for are TLS.  POW supports TLS, but tlslite
doesn't support POW.  I suspect that tlslite has some kind of driver
interface, since it already supports two crypto packages, so there may
be a simple answer here.  Not worth getting anywhere near this until
after making the jump to Twisted, as that might also affect this mess.
If the Python implementation ends up becoming the production version
of the engine rather than just a prototype, it might make sense to
revisit this, as the current mess of crypto libraries smells like an
accident waiting to happen.