Partly updated for new code, needs more work.

svn path=/myrpki.rototill/README; revision=3103

1 files changed, 152 insertions, 122 deletions

diff --git a/myrpki.rototill/README b/myrpki.rototill/README
index 3057fd0e..cc814021 100644
--- a/myrpki.rototill/README
+++ b/myrpki.rototill/README
@@ -12,16 +12,46 @@ fairly large programming project.  The tools in this directory attempt
 to fill that gap.
 
 This is a basic implementation of what a registry back end would need
-to use rpkid and friends.   These tools do not use every available
+to use rpkid and friends.  These tools do not use every available
 option, nor are they necessarily as efficient as possible.  Large
 registries will almost certainly want to roll their own tools, perhaps
 using these as a starting point.  Nevertheless, we hope that these
 tools will at least provide a useful example.
 
-The primary tools here consist of two Python programs: myrpki.py and
-myirbe.py.  The first is for use by any entity that needs resources
-allocated via the RPKI system, the second is for use by entities that
-actually run copies of rpkid and its several supporting programs.
+The primary tool here is a single command line Python program:
+myrpki.py.  myrpki has a number of commands, most of which are used
+for initial setup, some of which are used on an ongoing basis.  myrpki
+can be run either in an interactive mode or by passing a single
+command on the command line when starting the program; the former mode
+is intended to be somewhat human-friendly, the latter mode is useful
+in scripting, cron jobs, and automated testing.
+
+myrpki use has two distinct phases: setup and data maintenance.  The
+setup phase is primarily about constructing the "business PKI" (BPKI)
+certificates that the daemons use to authenticate CMS and HTTPS
+messages and obtaining the service URLs needed to configure the
+daemons.  The data maintenance phase is about configuring local data
+into the daemons.
+
+myrpki uses the OpenSSL command line tool for almost all operations on
+keys and certificates; the one exception to this is the comamnd which
+talks directly to the daemons, as this command uses the same
+communication libraries as the daemons themselves do.  The intent
+behind using the OpenSSL command line tool for everything else is to
+allow all the other commands to be run without requiring all the
+auxiliary packages upon which the daemons depend; this can be useful,
+eg, if one wants to run the back-end on a laptop while running the
+daemons on a server, in which case one might prefer not to have to
+install a bunch of unnecessary packages on the laptop.
+
+During setup phase myrpki generates and processes small XML messages
+which it expects the user to ship to and from its parents, children,
+etc via some out-of-band means (email, perhaps with PGP signatures,
+USB stick, we really don't care).  During data maintenance phase,
+myrpki does something similar with another XML file, to allow hosting
+of RPKI services; in the degenerate case where an entity is just
+self-hosting (ie, is running the daemons for itself, and only for
+itself), this latter XML file need not be sent anywhere.
 
 The basic idea here is that a user who has resources maintains a set
 of .csv files containing a text representation of the data needed by
@@ -29,29 +59,18 @@ the back-end, along with a configuration file containing other
 parameters.  The intent is that these be very simple files that are
 easy to generate either by hand or as a dump from relational database,
 spreadsheet, awk script, whatever works in your environment.  Given
-these files, the user then runs the myrpki.py script to extract the
-relevant information and encode everything about its back end state
-into a single .xml file, which the script writes out to disk.  The
-user then conveys this .xml file via some convenient means (PGP-signed
-mail, USB key, dog-sled) to the operator of the rpkid engine that will
-perform RPKI services on behalf of the user.
-
-The rpkid operator collects these .xml files from all the resource
-holders it hosts, and feeds them all into the myirbe.py script, which
-uses the data in the .xml files to populate the IRDB, create objects
-in rpkid and pubd via the left-right and publication protocols,
-etcetera.  The script rewrites its input .xml files to contain any
-updated information (eg, PKCS #10 requests for business signing
-context certificates), so that the .xml file once again contains
-everything that must be communicated between the rpkid operator and
-hosted resource holder.
-
-The rpkid operator ships the updated .xml back to the user, who then
-runs the myrpki.py script again to perform any necessary actions (eg,
-issuing business signing context certificates given the PKCS #10
-request sent by myirbe.py), resulting in another update to the .xml
-file, which the user then ships back to the rpkid operator.  This
-cycle repeats until nothing further needs to be changed.
+these files, the user then runs myrpki to extract the relevant
+information and encode everything about its back end state into an XML
+file, which can then be shipped to the appropriate other party.
+
+Many of the myrpki commands which process XML input write out a new
+XML file, either in place or as an entirely new file; in general,
+these files need to be sent back to the party that sent the original
+file.  Think of all this as a very slow packet-based communication
+channel, where each XML file is a single packet.  In setup phase,
+there's generally a single round-trip per setup conversation; in the
+data maintenance phase, the same XML file keeps bouncing back and
+forth between hosted entity and hosting entity.
 
 Note that, as certificates and CRLs have expiration and nextUpdate
 values, a low-level cycle of updates passing between resource holder
@@ -60,27 +79,14 @@ operation.  [The current version of these tools does not yet
 regenerate these expiring objects, but fixing this will be a
 relatively minor matter.]
 
-Since we assume that anybody who bothers to run rpkid is also a
-resource holder, myirbe.py and myrpki.py can use the same
-configuration file, and myirbe.py will run myrpki.py automatically if
-the [myrpki] section of the configuration file is present.
-
-The third important file in this system is the configuration file for
-myrpki.py and myirbe.py.  This contains a number of sections, some of
-which are for these scripts, others of which are for the OpenSSL
-command line tool, which these scripts use do most of the certificate
-work.  The examples/ subdirectory contains a commented version of the
+The third important kind of file in this system is the configuration
+file for myrpki.  This contains a number of sections, some of which
+are for myrpki, others of which are for the OpenSSL command line tool,
+still others of which are for the various RPKI daemon programs.  The
+examples/ subdirectory contains a commented version of the
 configuration file that explains the various parameters.
 
-myrpki.py deliberately does not use any libraries other than the ones
-that ship with Python 2.5; in particular, it does not require any of
-the other Python RPKI code.  This is intentional, to minimize
-portability issues for hosted resource holders.  It does require a
-reasonably current version of the OpenSSL command line tool, but the
-version that is built as a side effect of building the rcynic relying
-party tool is adequate if the system copy of this tool isn't.
-
-The .csv files read by myrpki.py can be anything that the Python "csv"
+The .csv files read by myrpki can be anything that the Python "csv"
 library understands.  By default, they're in tab-delimited format
 (because the author finds this easier to read than the comma-delimited
 format), but this can be changed to fit local needs.
@@ -94,17 +100,11 @@ separator, not one separator.  The upshot of all this is that
 attempting to make your columns line up prettily will not work as you
 expect, you will end up with too many cells, some of them empty.
 
-A number of the fields in the configuration or CSV files involve
-certificates.  Some of these are built automatically, others must be
-imported so that the scripts can cross-certify them.  The certificates
-you need to import are all self-signed BPKI trust anchor certificates
-generated by other entities; you import them by specifying the name of
-a file where you stored the BPKI certificate in question (in OpenSSL
-"PEM" format).
-
 Keep reading, and don't panic.
 
-The default configuration file name is myrpki.conf.
+The default configuration file name is myrpki.conf.  You can change
+this using the "-c" option when invoking myrpki, or by setting the
+environment variable MYRPKI_CONF.
 
 See examples/myrpki.conf for details on the variables that you can
 (and in some cases must) set.
@@ -113,17 +113,13 @@ See examples/*.csv for commented examples of the several CSV files.
 Note that the comments themselves are not legal CSV, they're just
 present to make it easier to understand the examples.
 
-GETTING STARTED -- OVERVIEW
 
-As explained above, the two basic programs are myrpki.py (for resource
-holders) and myirbe.py (for rpkid operators); myirbe.py runs myrpki.py
-automatically for a rpkid operator's own resources if myirbe.py finds
-a [myrpki] section in its configuration file.
+GETTING STARTED -- OVERVIEW
 
-Which process you need to follow to get started depends on whether you
-are running rpkid yourself or will be hosted by somebody else.  We
-call the first case "self-hosted", because the software treats running
-rpkid to handle resources that you yourself hold as if you are an rpkid
+Which process you need to follow depends on whether you are running
+rpkid yourself or will be hosted by somebody else.  We call the first
+case "self-hosted", because the software treats running rpkid to
+handle resources that you yourself hold as if you are an rpkid
 operator who is hosting an entity that happens to be yourself.
 
 "$top" in the following refers to wherever you put the
@@ -132,6 +128,97 @@ install" targets, this will be some system directory or another; for
 now, it's wherever you checked out a copy of the code from the
 subversion repository or unpacked a tarball of the code.
 
+Most of the setup process looks the same for any resource holder,
+regardless of whether they are self-hosting or not.  The differences
+come in the data maintenence phase.
+
+The steps needed during setup phase are:
+
+- Write a configuration file (copy $top/myrpki/examples/myrpki.conf
+  and edit as needed).  You need to configure the [myrpki] section; in
+  theory, the rest of the file should be ok as it is, at least for
+  simple use.
+
+- Initialization ("initialize" command).  This creates the local BPKI
+  and other data structures that can be constructed just based on
+  local data such as the config file.  Other than some internal data
+  structures, the main output of this step is the "identity.xml" file,
+  which is used as input to later stages.
+
+  In theory it should be safe to run the "initialize" command more
+  than once, in practice this has not (yet) been tested.
+
+- Send (email, USB stick, carrier pigeon) identity.xml to each of your
+  parents.  This tells each of your parents what you call yourself,
+  and supplies each parent with a trust anchor for your
+  resource-holding BPKI.
+
+- Each of your parents runs the "configure_child" command, giving the
+  identity.xml you supplied as input.  This registers your data with
+  the parent, including BPKI cross-registration, and generates a
+  return message containing your parent's BPKI trust anchors, a
+  service URL for contacting your parent via the "up-down" protocol,
+  and (usually) either an offer of publication service (if your parent
+  operates a repository) or a referral from your parent to whatever
+  publication service your parent does use.  Referrals include a
+  CMS-signed authorization token that the repository operator can use
+  to determine that your parent has given you permission to home
+  underneath your parent in the publication tree.
+
+- Each of your parents sends (...) back the response XML file
+  generated by the "configure_child" command.
+
+- You feed the response message you just got into myrpki using the
+  "configure_parent" command.  This registers the parent's information
+  in your database, including BPKI cross-certification, and processes
+  the repository offer or referral to generate a publication request
+  message.
+
+- You send (...) the publication request message to the repository.
+  The <contact_info/> element in the request message should (in
+  theory) provide some clue as to where you should send this.
+
+- The repository operator processes your request using myrpki's
+  "configure_publication_client" command.  This registers your
+  information, including BPKI cross-certification, and generates a
+  response message containing the repository's BPKI trust anchor and
+  service URL.
+
+- Repository operator sends (...) the publication confirmation message
+  back to you.
+
+- You process the publication confirmation message using myrpki's
+  "configure_repository" command.
+
+At this point you should, in theory, have established relationships,
+exchanged trust anchors, and obtained service URLs from all of your
+parents and repositories.  The last setup step is establishing a
+relationship with your RPKI service host, if you're not self-hosted,
+but as this is really just the first message of an ongoing exchange
+with your host, it's handled by the data maintenance commands.
+
+The two commands used in data maintenence phase are
+"configure_resources" and "configure_daemons".  The first is used by
+the resource holder, the second is used by the host.  In the
+self-hosted case, it is not necessary to run "configure_resources" at
+all, myrpki will run it for you automatically.
+
+GETTING STARTED -- CONFIGURATION FILE
+
+The sample configuration file should, in theory, be much simpler to
+use than in earlier versions of this code.  The sample configuration
+uses a simple macro-expansion mechanism to place all of the
+configuration data you need to touch into the [myrpki] section; the
+rest of the configuration file is for the various other tools, and is
+entirely configured via references to the values defined in the
+[myrpki] section.
+
+
+
+[[[[**** CONTINUE HERE ****]]]]
+
+
+
 GETTING STARTED -- HOSTED CASE
 
 The basic steps involved in getting started for a resource holder who
@@ -601,60 +688,3 @@ environment variable, see the ld.so man page for details.  This is a
 relatively minor variation on the usual build issues for shared
 libraries, it's just annoying because shared libraries should not be
 needed here and would not be if not for this GNU linker issue.
-
-
-
-Sketch towards a simple description of the BPKI (sic).
-
-This started out as notes to myself during a redesign, and badly needs
-rewriting.
-
-Hosted (myrpki) entity needs:
-
-- Self-signed BPKI root (doesn't really need to be self-signed, nobody
-  else will care, but self-signed is simplest for our purposes).  This
-  is what we've been calling the "self" cert in testbed.py.
-
-- BSC EE issued by self-signed root.
-
-- Cross-certs of every foreign entity (parent, child, or pubd): these
-  are CA certs with pathLenConstraint 0.  Input for this cross-cert is
-  self-signed (or whatever) from foreign entity, output is
-  pathLenConstraint 0 CA cert issued by myrpki entity's own
-  self-signed root.
-
-Hosting rpkid (myirbe) needs:
-
-- Self-signed BPKI root
-
-- BSC EE certs for rpkid, irdbd, irbe_cli, etc
-
-- For each hosted entity (including self-hosting):
-
-  Cross-cert of hosted entity's root, issued by rpkid root: CA cert
-  with pathLenConstraint 1
-
-  In theory that's all that's required, everything else is handled
-  through the hosted entity's cert chain.
-
-pubd needs:
-
-- Self signed root (might share with rpkid but let's keep it separate
-  conceptually)
-
-- BSC EE certs for pubd and irbe_cli
-
-- For each client entity of pubd:
-
-  Cross-cert of client entity's self cert (pathLenConstraint 0).
-
-  This should allow pubd to verify clients' BSC EE certs without
-  getting into transitive CA relationships.
-
-rootd (when applicable at all) needs:
-
-- Self-signed root
-
-- BSC EE cert for talking up-down (server) with one and only child
-
-- Cross-cert (pathLenConstraint 0) of one and only child's self cert.