$Id$ -*- Text -*-

Python RPKI production tools.

Requires Python 2.5.

External Python packages required:

- lxml, which in turn requires the libxml2 C libraries.

  FreeBSD: /usr/ports/devel/py-lxml

- MySQLdb, which in turn requires MySQL client and server.  I'm
  testing with MySQL 5.1.

  FreeBSD: /usr/ports/databases/py-MySQLdb

- TLSLite, which pulls in other crypto packages.

  FreeBSD: /usr/ports/security/py-tlslite

- Cryptlib, at the moment just to support TLSlite but may end up using
  it for other things later.

  FreeBSD: /usr/ports/security/cryptlib

  ...but the FreeBSD port doesn't (yet?) install the Python bindings,
  sigh, so at the moment you have to do that by hand:

  # cd /usr/ports/security/cryptlib
  # make install
  # cd work/bindings
  # python setup.py install
  # cd ../..
  # make clean

- Eventually I expect that this will require an event-handling package
  like Twisted, but I'm not there yet.

- The testpoke tool (up-down protocol command line test client) also
  uses PyYAML, mostly for compatability with APNIC's equivalent tool.

  FreeBSD: /usr/ports/devel/py-yaml

We also use a hacked copy of the Python OpenSSL Wrappers (POW)
package, but our copy has enough modifications that it's expanded in
the Subversion tree.  Depending on how this all works out, I may end
up splitting the POW.pkix module out of the POW package and using it
with Cryptlib, as the POW.pkix package is 98% about doing ASN.1 in
pure Python and only 2% about any kind of crypto.



Current TO DO list:

- Client side of expiration: how do we decide when to ask for a new
  cert from our parent?  Tunable (preference, if so)?   Waiting for
  the old one to die is not good.  Reissue automatically when IRDB
  valid_until changes?

- Parent side of revocation (child_cert objects) and CRL generation
  implemented but not yet tested.

- Child side of revocation...Common Management Tasks page in the APNIC
  Wiki shows some states where revocation is triggered by the child
  after a delay.  Other text in Common Management Tasks suggests that
  ca_detail also needs deferred transitions from pending to active.
  So added a deferred state transition timestamp to ca_detail object:

  How to model this?  An enumerated state value (already have that,
  may need to rename states) and a timestamp (which may be NULL) for
  next scheduled transition?  Is the next state always predictable
  from the current state or do we also need a next-state field?

    state     := pending | active | deprecated | revoked
    timestamp := null | <time of next transition>

  Do we really need the revoked state?  That is, once we've reached
  that state, do we have any further use for the ca_detail in our
  database, or do we just delete it (and its little dog too...)?  If
  we do need the revoked state, I guess the timer becomes the delay
  until we can get rid of it entirely, or something like that.

  For that matter, how do we, as child, even find out that a cert has
  been revoked?

  a) We asked to have it revoked, duh.

  b) Parent reissued with same resource class and key, revoking the
     old cert (oversize, or something).  We have to detect this when
     processing <list_response/> and probably also <issue_response/>,
     and perform immediate reissue to any affected children, because
     the old cert is no good anymore.

  In either case we're done with the old cert once it's been revoked.

- Publication protocol and implementation thereof.   Defer until core
  functionality in the main engine is done.

  As an interim measure, hack some kind of stub publication (not real
  protocol yet, just dump to local filesystem so can see outputs and
  maybe rcynic against them); this is a stop-gap to let me concentrate
  on the main engine and defer work on the publication protocol and
  engine.

- Publication hooks everywhere - need not wait for protocol, can just
  log what would happen for now, or write to local file store (perhaps
  even in a form that we can use with rcynic as a relying party).
  Hooks for this go into:

  - Cert publication

  - CRL publication

  - Manifest publication

  - Withdrawal of any of the above

  Consolidate generation of filenames and URIs into methods, duh.

  (tags-search "\\.\\(publish\\|withdraw\\)(\\|\\+ \"\\.")

- Logging subsystem, including syslog support.

- Child batch processing loop, eg, regeneration or removal of expired
  certs, CRL update, manifest update, etc.  This should probably be an
  iteration over CA objects, as the CA is the actor in pretty much
  everything that might need to be done.

  Figuring out whether to regenerate or remove expired certs requires
  some of the same data as CRL generation.

  - Code to clean up expired certs

  - Code to revoke certs -- need to sort out when we do this
    automatically vs waiting for explicit revoke PDU from child

  - Code to generate CRLs

- Subsetting (req_* attributes in up-down protocol)

- Haven't done anything about db.commit() and db.rollback() yet, for
  that matter haven't yet whacked MySQL to enable those features.

- In theory, all access to object data attributes ought to be through
  accessor methods so that the .set() method can mark teh object as
  SQL-dirty automagically.  Not done yet.  One way of hiding the
  grotty bits here might be to make attributes look like elements of a
  dict(), ie, implement __getitem__() and __setitem__() methods,
  probably along with the other "container type" special methods for
  completeness.  An even easier way to do this would be to subclass
  dict() and just customize __setitem__().

  But all of this assumes that we really want automagic SQL-dirty, and
  I'm less convinced of that now than when I made the above note.

- Test with larger data set -- Tim gave me plenty of data and I have
  the low-level tools, just haven't written the glue logic to create
  child objects for all the entities in the IRDB, poll on behalf of
  each of them, and check the result for sanity

- Need to figure out how we're going to handle "root" case (IANA/RIR
  self-signed resource cert issuing to its children).  Right now this
  is a separate little program, which is nice for testing but feels
  wrong.  As Randy puts it, this is kind of a special case of a
  parent, although we might want to represent it differently.  If this
  were really only IANA and the RIRs, handling it as a special case
  separate program might make sense, but anybody who wants to certify
  private address space is going to have this problem, and the
  separate daemon approach just feels wrong for that.

  If it's not a separate daemon, will need left-right protocol support
  to configure whatever it is we're going to configure, with all the
  usual private key hygiene issues.  This probably implies a level of
  indirection, eg, the self-signed cert is generated in the IRBE, the
  RPKI engine generates PKCS#10 for a working cert to be issued by the
  self-signed cert (perhaps with RFC 3779 inheritance for everything,
  to keep it small), so that the RPKI engine never needs to hold the
  private key for the root.

  Deferred for the moment, not sure for how long.

Once this lot is done we'll be close to something that shows at least
the basics of normal operation, albiet in a form that's not yet usable
in production.

Follow-up after that will be getting rid of remaining synchronous code
(make daemon fully event-driven, except perhaps for SQL queries),
address rollback, commit, and other data integrity issues, and see how
well the resulting code handles hosting (multiple self objects in same
daemon).

Somewhere along the way I'll need to update to the new model of trust
anchors we ended up with in Amsterdam, first step for which will
involve writing it down (well, RobK was supposed to do that, but I was
supposed to convert some pencil sketches into graphviz for him so
we're both lame on this so far).  I don't think this results in major
changes, probably a few extra cert fields in the self object which we
then need to toss into the rpki.x509.X509_chain objects before
verifying CMS or TLS, and perhaps the existing TA fields in various
objects become pairs of certs instead of a single TA, but this is
mostly just generalization and reuse of existing code, no bold new
adventures.

At some point we need to do performance testing.  All of our crypto is
done internally except for CMS, for which we still call the OpenSSL
CLI tool.  The simplest solution would probably be to extend POW to
support CMS; cryptlib might be more entertaining solution, but am not
sure it's worth the time sink.

Although it may not be worth the effort, it'd be nice to get back to
only one crypto library.  As far as I know the only things we're using
M2Crypto or cryptlib for are TLS.  POW supports TLS, but tlslite
doesn't support POW.  I suspect that tlslite has some kind of driver
interface, since it already supports two crypto packages, so there may
be a simple answer here.  Not worth getting anywhere near this until
after making the jump to Twisted, as that might also affect this mess.
If the Python implementation ends up becoming the production version
of the engine rather than just a prototype, it might make sense to
revisit this, as the current mess of crypto libraries smells like an
accident waiting to happen.