First pass updating this for the current software.

svn path=/myrpki.rototill/README; revision=3113

1 files changed, 136 insertions, 365 deletions

diff --git a/myrpki.rototill/README b/myrpki.rototill/README
index cc814021..945ca18a 100644
--- a/myrpki.rototill/README
+++ b/myrpki.rototill/README
@@ -134,61 +134,64 @@ come in the data maintenence phase.
 
 The steps needed during setup phase are:
 
-- Write a configuration file (copy $top/myrpki/examples/myrpki.conf
-  and edit as needed).  You need to configure the [myrpki] section; in
-  theory, the rest of the file should be ok as it is, at least for
-  simple use.
-
-- Initialization ("initialize" command).  This creates the local BPKI
-  and other data structures that can be constructed just based on
-  local data such as the config file.  Other than some internal data
-  structures, the main output of this step is the "identity.xml" file,
-  which is used as input to later stages.
-
-  In theory it should be safe to run the "initialize" command more
-  than once, in practice this has not (yet) been tested.
-
-- Send (email, USB stick, carrier pigeon) identity.xml to each of your
-  parents.  This tells each of your parents what you call yourself,
-  and supplies each parent with a trust anchor for your
-  resource-holding BPKI.
-
-- Each of your parents runs the "configure_child" command, giving the
-  identity.xml you supplied as input.  This registers your data with
-  the parent, including BPKI cross-registration, and generates a
-  return message containing your parent's BPKI trust anchors, a
-  service URL for contacting your parent via the "up-down" protocol,
-  and (usually) either an offer of publication service (if your parent
-  operates a repository) or a referral from your parent to whatever
-  publication service your parent does use.  Referrals include a
-  CMS-signed authorization token that the repository operator can use
-  to determine that your parent has given you permission to home
-  underneath your parent in the publication tree.
-
-- Each of your parents sends (...) back the response XML file
-  generated by the "configure_child" command.
-
-- You feed the response message you just got into myrpki using the
-  "configure_parent" command.  This registers the parent's information
-  in your database, including BPKI cross-certification, and processes
-  the repository offer or referral to generate a publication request
-  message.
-
-- You send (...) the publication request message to the repository.
-  The <contact_info/> element in the request message should (in
-  theory) provide some clue as to where you should send this.
-
-- The repository operator processes your request using myrpki's
-  "configure_publication_client" command.  This registers your
-  information, including BPKI cross-certification, and generates a
-  response message containing the repository's BPKI trust anchor and
-  service URL.
-
-- Repository operator sends (...) the publication confirmation message
-  back to you.
-
-- You process the publication confirmation message using myrpki's
-  "configure_repository" command.
+0) Write a configuration file (copy $top/myrpki/examples/myrpki.conf
+   and edit as needed).  You need to configure the [myrpki] section;
+   in theory, the rest of the file should be ok as it is, at least for
+   simple use.  You also need to create (either by hand or by dumping
+   from a database, spreadsheet, whatever) the CSV files describing
+   prefixes and ASNs you want to allocate to your children and ROAs
+   you want created.
+
+1) Initialization ("initialize" command).  This creates the local BPKI
+   and other data structures that can be constructed just based on
+   local data such as the config file.  Other than some internal data
+   structures, the main output of this step is the "identity.xml" file,
+   which is used as input to later stages.
+
+   In theory it should be safe to run the "initialize" command more
+   than once, in practice this has not (yet) been tested.
+
+2) Send (email, USB stick, carrier pigeon) identity.xml to each of your
+   parents.  This tells each of your parents what you call yourself,
+   and supplies each parent with a trust anchor for your
+   resource-holding BPKI.
+
+3) Each of your parents runs the "configure_child" command, giving the
+   identity.xml you supplied as input.  This registers your data with
+   the parent, including BPKI cross-registration, and generates a
+   return message containing your parent's BPKI trust anchors, a
+   service URL for contacting your parent via the "up-down" protocol,
+   and (usually) either an offer of publication service (if your parent
+   operates a repository) or a referral from your parent to whatever
+   publication service your parent does use.  Referrals include a
+   CMS-signed authorization token that the repository operator can use
+   to determine that your parent has given you permission to home
+   underneath your parent in the publication tree.
+
+4) Each of your parents sends (...) back the response XML file
+   generated by the "configure_child" command.
+
+5) You feed the response message you just got into myrpki using the
+   "configure_parent" command.  This registers the parent's information
+   in your database, including BPKI cross-certification, and processes
+   the repository offer or referral to generate a publication request
+   message.
+
+6) You send (...) the publication request message to the repository.
+   The <contact_info/> element in the request message should (in
+   theory) provide some clue as to where you should send this.
+
+7) The repository operator processes your request using myrpki's
+   "configure_publication_client" command.  This registers your
+   information, including BPKI cross-certification, and generates a
+   response message containing the repository's BPKI trust anchor and
+   service URL.
+
+8) Repository operator sends (...) the publication confirmation message
+   back to you.
+
+9) You process the publication confirmation message using myrpki's
+   "configure_repository" command.
 
 At this point you should, in theory, have established relationships,
 exchanged trust anchors, and obtained service URLs from all of your
@@ -205,77 +208,43 @@ all, myrpki will run it for you automatically.
 
 GETTING STARTED -- CONFIGURATION FILE
 
-The sample configuration file should, in theory, be much simpler to
-use than in earlier versions of this code.  The sample configuration
-uses a simple macro-expansion mechanism to place all of the
-configuration data you need to touch into the [myrpki] section; the
-rest of the configuration file is for the various other tools, and is
-entirely configured via references to the values defined in the
-[myrpki] section.
-
-
-
-[[[[**** CONTINUE HERE ****]]]]
-
-
+The current sample configuration file should, in theory, be much
+simpler to use than in earlier versions of this code.  The sample
+configuration uses a simple macro-expansion mechanism to place all of
+the configuration data you need to touch into the [myrpki] section;
+the rest of the configuration file is for the various daemons and
+other tools, and is entirely configured via references to the values
+defined in the [myrpki] section.
 
 GETTING STARTED -- HOSTED CASE
 
 The basic steps involved in getting started for a resource holder who
 is being hosted by somebody else are:
 
-1) Obtain contact information and BPKI trust anchors from RPKI parents
-   and an RPKI publication service (see below for details).
-
-2) Write a configuration file (copy $top/myrpki/examples/myrpki.conf
-   and edit as needed).  You can skip the sections associated with the
-   various daemons and their runtime control tools ([myirbe], [rpkid],
-   [irdbd], [pubd], [rootd], [irbe_cli]).  You *do* need to configure
-   the [myrpki] section.
-
-3) Using $top/myrpki/examples/*.csv as a guide, create a set of CSV
-   files representing RPKI parents, RPKI children, resources to be
-   assigned to RPKI children, and ROAs to be generated once the
-   necessary RPKI certificates are available.  Most of these CSV files
-   can be empty while first getting started, the only file that
-   absolutely must be populated is the file describing parents.
-
-   You may choose to place your configuration file (which we will
-   refer to here as myrpki.conf) and your CSV files in their own
-   directory.  The software doesn't really care.  If you use absolute
-   names for all the filename entries in the configuration file and
-   CSV files, you can put the files wherever you like; if you use
-   relative names, they will be interpreted relative to the directory
-   in which you run the program that reads the file.
-
-   [At some future date we may provide a default directory for
-   relative filenames such as /usr/local/etc/rpki, but the above
-   description holds for now.]
-
-4) Run myrpki.py to generate a BPKI trust anchor and collect all the
-   data from the configuration file, CSV files, and newly created BPKI
-   into a single XML file which can be shipped to the rpkid operator
-   who is hosting your resources.
-
-5) Send the XML file generated in step (4) to your rpkid operator.
-
-6) Wait for your rpkid operator to ship you back an updated XML file
+a) Run through steps (0)-(9), above.
+
+b) Run the configure_resources command to generate myrpki.xml.
+
+c) Send myrpki.xml to the rpkid operator who will be hosting you.
+
+d) Wait for your rpkid operator to ship you back an updated XML file
    containing a PKCS #10 certificate request for the BPKI signing
    context (BSC) created by rpkid.
 
-7) Run myrpki.py again with the XML file received in step (6), to
-   issue the BSC certificate and update the XML file again to contain
-   the newly issued BSC certificate.
+e) Run configure_resources again with the XML file received in step
+   (d), to issue the BSC certificate and update the XML file again to
+   contain the newly issued BSC certificate.
 
-8) Send the updated XML file back to your rpkid operator.
+f) Send the updated XML file back to your rpkid operator.
 
 At this point you're done with initial setup.  You will need to run
-myrpki.py again whenever you make any changes to your configuration
-file or CSV files.  [Once myrpki.py knows how to update BPKI CRLs, you
-will also need to run myrpki.py periodically to keep your BPKI CRLs up
-to date.]  Any time you run myrpki.py, you should send the updated XML
-file to your rpkid operator, who will [generally?] send you a further
-updated XML file in response.
+configure_resources again whenever you make any changes to your
+configuration file or CSV files.  [Once myrpki knows how to update
+BPKI CRLs, you will also need to run configure_resources periodically
+to keep your BPKI CRLs up to date.]  Any time you run
+configure_resources myrpki, you should send the updated XML file to
+your rpkid operator, who will [generally?] send you a further updated
+XML file in response.
 
 GETTING STARTED -- SELF-HOSTED CASE
 
@@ -284,67 +253,24 @@ resource holder (that is, a resource holder that runs its own copy of
 rpkid) are the same as in the hosted case above; after that the
 process diverges.
 
-[As of the time at which these instructions were written, it had
-become clear that there really should be an additional setup script
-which automates much of the following.  That script hasn't been
-written yet, so for the moment this documents the setup process as it
-stands now.  Once that setup script has been written, these
-instructions will be updated to match.  In the meantime, please accept
-the author's apologies for the tedious nature of the current setup
-process.]
-
 The [current] steps are:
 
-1) Obtain contact information and BPKI trust anchors from RPKI parents
-   and an RPKI publication service (see below for details).
-
-2) Write a configuration file (copy examples/myrpki.conf and edit as
-   needed).  You need to configure the [myrpki] and [myirbe] sections
-   as well as the sections associated with the daemons you will be
-   running ([rpkid], [irdbd], [irbe_cli]).  You only need to configure
-   the [pubd] section if you intend to run your own publication
-   service: in general this is not recommended, because each
-   additional publication service in the RPKI universe places a small
-   additional burden on every relying party, since every relying party
-   has to download data from every publication service.  In general
-   it's better to use an existing publication service operated by
-   somebody else (eg, your RPKI parent) if you can.  In general most
-   cases you can leave the [rootd] section alone, as in most cases you
-   should not be running rootd.
-
-3) Using $top/myrpki/examples/*.csv as a guide, create a set of CSV
-   files representing RPKI parents, RPKI children, resources to be
-   assigned to RPKI children, and ROAs to be generated once the
-   necessary RPKI certificates are available.  Most of these CSV files
-   can be empty while first getting started, the only file that
-   absolutely must be populated is the file describing parents.
-
-   You may choose to place your configuration file (which we will
-   refer to here as myrpki.conf) and your CSV files in their own
-   directory.  The software doesn't really care.  If you use absolute
-   names for all the filename entries in the configuration file and
-   CSV files, you can put the files wherever you like; if you use
-   relative names, they will be interpreted relative to the directory
-   in which you run the program that reads the file.
-
-   [At some future date we may provide a default directory for
-   relative filenames such as /usr/local/etc/rpki, but the above
-   description holds for now.]
-
-4) See rpkid/doc/Installation, and follow the basic installation
+a) See rpkid/doc/Installation, and follow the basic installation
    instructions there to build the RFC-3779-aware OpenSSL code and
    associated Python extension module.
 
-5) Next, you need to set up the MySQL databases that rpkid et al will
+b) Run through steps (0)-(9), above.
+
+c) Next, you need to set up the MySQL databases that rpkid et al will
    use.  The MySQL database, username, and password values all need to
    match the ones you specified in myrpki.conf.  There are two
    different ways you can do this:
 
-   a) You can use the setup-sql.py script, which prompts you for your
+   i) You can use the setup-sql.py script, which prompts you for your
       MySQL root password then attempts to do everything else
       automatically using values from myrpki.conf; or
 
-   b) You can do it manually.
+   ii) You can do it manually.
 
    The first approach is simple:
 
@@ -387,18 +313,7 @@ The [current] steps are:
    mysql> COMMIT;
    mysql> quit
 
-6) Run myirbe.py -b to set up the initial BPKI structure needed to run
-   your daemons:
-
-   $ python $top/myrpki/myirbe.py -b
-
-   The -b option tells myrpki.py that you want it to stop after the
-   initial BPKI setup, regardless of whether it thinks this is
-   necessary.  If you have not done this before it should tell you
-   that it has updated the BPKI and that you need to (re)start daemons
-   now.
-
-7) If you are running your own publication repository (that is, if you
+d) If you are running your own publication repository (that is, if you
    are running pubd), you will also need to set up an rsyncd server or
    configure your existing one to serve pubd's output.  There's a
    sample configuration file in $top/myrpki/examples/rsyncd.conf, but
@@ -406,23 +321,21 @@ The [current] steps are:
    running rsyncd for other purposes.  See the rsync(1) and
    rsyncd.conf(5) manual pages for more details.
 
-8) Start the daemons.  You can use $top/myrpki/start-servers.py to do
+e) Start the daemons.  You can use $top/myrpki/start-servers.py to do
    this, or write your own script.
 
    If you intend to run pubd, you should make sure that the directory
-   you specified as publication-base in the [pubd] section exists and
+   you specified as publication_base_directory exists and
    is writable by the userid that will be running pubd, and should
    also make sure to start rsyncd.
 
-9) Run myirbe.py again, twice, this time with no arguments.
+f) Run myrpki's configure_daemons command, twice, with no arguments.
 
-   $ python $top/myrpki/myirbe.py
-   $ python $top/myrpki/myirbe.py
-
-   The reason for running myirbe.py twice at this point is explained
-   in the Introduction section, above; in brief, the first run sets up
-   almost everything, but a second pass is required to generate the
-   BSC certificate.
+   You need to run the command twice because myrpki has to ask rpkid
+   to create a keypair and generate a certification request for the
+   BSC.  The first pass does this, the second processes the
+   certification request, issues the BSC, and loads the result into
+   rpkid.  [Yes, we could automate this somehow, if necessary.]
 
 At this point, if everything went well, rpkid should be up,
 configured, and starting to obtain resource certificates from its
@@ -430,193 +343,51 @@ parents, generate CRLs and manifests, and so forth.  At this point you
 should go figure out how to use the relying party tool, rcynic: see
 $top/rcynic/README if you haven't already done so.
 
-If and when you change your CSV files, you should run myirbe.py again
-to feed the changes into the daemons.
+If and when you change your CSV files, you should run
+configure_daemons again to feed the changes into the daemons.
 
 GETTING STARTED -- HOSTING CASE
 
 If you are running rpkid not just for your own resources but also to
 host other resource holders (see "HOSTED CASE" above), your setup will
 be almost the same as in the self-hosted case (see "SELF-HOSTED CASE",
-above), with one procedural change: you will need to tell myirbe.py to
-process the XML files produced by the resource holders you are
-hosting.  You do this by specifying the names of all those XML files
-on myirbe's command line.  So, if you are hosting two friends, Alice
-and Bob, then, everywhere the instructions for the self-hosted case
-say to run myirbe.py with no arguments, you will instead run it with
-the names of Alice's and Bob's XML files:
-
-  $ python $top/myrpki/myirbe.py alice.xml bob.xml
-
-Note that myirbe.py sometimes modifies these XML files, in which case
-it will write them back to the same filenames.  While it is possible
-to figure out the set of circumstances in which myirbe.py will modify
-XML files (at present, this only happens when myirbe.py has to ask
-rpkid to create a new BSC keypair and PKCS #10 certificate request),
-it may be easiest just to ship back an updated copy of the XML file
-after every you run myirbe.py.
+above), with one procedural change: you will need to tell
+configure_daemons to process the XML files produced by the resource
+holders you are hosting.  You do this by specifying the names of all
+those XML files on as arguments to the configure_daemons command.  So,
+if you are hosting two friends, Alice and Bob, then, everywhere the
+instructions for the self-hosted case say to run configure_daemons
+with no arguments, you will instead run it with the names of Alice's
+and Bob's XML files as arguments.
+
+Note that configure_daemons sometimes modifies these XML files, in
+which case it will write them back to the same filenames.  While it is
+possible to figure out the set of circumstances in which this will
+happen (at present, only when myrpki has to ask rpkid to create a new
+BSC keypair and PKCS #10 certificate request), it may be easiest just
+to ship back an updated copy of the XML file after every you run
+configure_daemons.
 
 GETTING STARTED -- "PURE" HOSTING CASE
 
 In general we assume that anybody who bothers to run rpkid is also a
-resource holder, but the software does not insist on this.  If you are
-running rpkid solely for others and have no resources of your own, the
-process is almost identical to the "HOSTING CASE", above.  The one
-change is that you should *not* have a [myrpki] section in your
-configuration file.
-
-A (perhaps) slightly-more-plausible use for this capability would be
-if you are an rpkid-running resource holder who wants for some reason
-to keep the resource-holding side of your operation completely
-separate from the rpkid-running side of your operation.  This is
-essentially the pure-hosting model, just with an internal hosted
-entity within a different part of your own organization.
-
-DATA YOU NEED FROM YOUR RPKI PARENT AND PUBLICATION SERVICE
-
-In order to connect to your RPKI parent, you will need to supply your
-BPKI trust anchor to your parent and obtain four pieces of data from
-your parent.
-
-Assuming that you are using something resembling the default
-configuration, your BPKI trust anchor will be bpki.myrpki/ca.cer.
-This is an OpenSSL "PEM" format file.  You will need to provide this
-to your RPKI parent.
-
-The data you need from your parent are:
-
-- The service URL for your entry point into your parent's rpkid.
-  Typically this will be a URL of the form:
-
-  https://example.org:port/up-down/parenthandle/myhandle
-
-  where "example.org" and "port" are the DNS name and TCP port of your
-  parent's rpkid service, "parenthandle" is your parent's name
-  (handle) for itself, and "myhandle" is your parent's name (handle)
-  for you;
-
-- Your parent's BPKI trust anchor for its resource-holding persona
-  (the entity represented by "parenthandle", above);
-
-- Your parent's BPKI trust anchor for daemons it operates; and
-
-- The handle by which your parent refers to you in its database,
-  generally the same as "myhandle" in the service URL.
-
-The need for two separate BPKI trust anchors for your parent is due to
-a limitation of the HTTPS protocol; recent extensions to TLS provide a
-way to work around this limitation, but at this point in time rpkid
-can't assume support for the TLS extension in question.  Roughly
-speaking, the first BPKI trust anchor corresponds to the your parent
-as a resource-holding entity, while the second corresponds to your
-parent as an rpkid-operating entity.
-
-These four data correspond, in order, to the second, third, fourth,
-and fifth columns in your parents.csv file.  In most cases you will
-have only one parent, so there will be only one line in that file.
-
-The first field in the parents.csv file is your name for your parent,
-which can be any name you like so long as it doesn't conflict with
-your name for another parent.
-
-The sixth field in the parents.csv file determines the base rsync URI
-for objects signed by certificates issued by this parent.  If you are
-using an external publication service (recommended), your parent must
-supply this URI as well; a typical value would be
-rsync://example.org/Dad/Me/ or rsync://example.org/Grandma/Dad/Me/.
-
-If you are running your own copy of pubd, this URI should point to the
-directory that corresponds to the publication-base setting in the
-[pubd] section of your configuration file.
-
-If you are using an external publication service (which might be your
-parent, grandparent, or any ancestor all the way up to the root), your
-publication service will also need to tell you:
-
-- The service URL for the publication service (pubd_base parameter in
-  [myirbe] section of your configuration file);
-
-- The publication service's name for you (repository_handle field in
-  [myrpki] section of your configuration file); and
-
-- The BPKI trust anchor for the publication service
-  (repository_bpki_certificate field in [myrpki] section of your
-  configuration file).
-
-Note that the first of these three parameters only applies if you are
-running rpkid, while the second and third apply even if your resources
-are hosted on somebody else's rpkid.  In effect, this means that all
-the entities sharing a single rpkid must also share a single
-publication service.  This is a restriction of the myrpki/myirbe
-software, not rpkid itself, so it could be removed if there were a
-strong need to do so, but given that each additional publication
-service imposes a small additional burden on every relying party in
-the world, we do not view this restriction as a problem.
-
-DATA YOU NEED TO GIVE YOUR RPKI CHILDREN AND USERS OF YOUR PUBLICATION SERVICE
-
-First, read the previous section describing what children and
-publication clients expect to receive.
-
-- The service URL for your rpkid will be an HTTPS URL of the form 
-
-  https://example.org:port/up-down/yourhandle/childhandle
-
-  where "example.org" and "port" are the DNS name and TCP port of your
-  rpkid service ([rpkid] section of your configuration file),
-  "yourhandle" is the handle parameter from the [myrpki] section of
-  your configuration file, and "childhandle" is this child's handle as
-  it appears in the first columns of your children.csv, asns.csv, and
-  prefixes.csv files;
-
-- The BPKI trust anchor for your resource-holding persona is your
-  bpki.myrpki/ca.cer;
-
-- The BPKI trust anchor for daemons you operate is your
-  bpki.myirbe/ca.cer; and
-
-- The handle by which you refer to your child is the same as
-  "childhandle", above.
-
-If you are operating a publication service, you will also need to
-supply:
-
-- Your pubd service URL, which will be an HTTPS URL of the form
-
-  https://example.org:port/
-
-  where "example.org" and "port" are the server-host and server-port
-  parameters from the [pubd] section of your configuration file;
-
-- Your name for this publication client, which is the first column of
-  your pubclients.csv file (note that this can be a structured name
-  using "/" characters as a hierarchy delimiter); and
-
-- The BPKI trust anchor for the daemons you operate
-  (bpki.myirbe/ca.cer).
-
-Note that, if you are operating pubd, it's best for relying parties if
-your children's publication points are underneath yours within the
-publication hierarchy, to allow rsync to check for updates as
-efficiently as possible.  pubd's support for hierarchical client
-handles is intended to simplify this: if you have a child Alice, who
-has children Bob and Bill, and you, your children, and your
-grandchildren will all be using your publication service, you might
-assign <client_handle> and <sia_base> parameters (first and third
-fields in pubclients.csv) as follows:
-
-Me		rsync://rpki.example.org/Me/
-Me/Alice	rsync://rpki.example.org/Me/Alice/
-Me/Alice/Bob	rsync://rpki.example.org/Me/Alice/Bob/
-Me/Alice/Bill	rsync://rpki.example.org/Me/Alice/Bill/
-
-Note that you will need trust anchors for your children and any
-publication clients.  In both cases the trust anchor you need is the
-child's or client's resource-holding BPKI trust anchor
-(bpki.myrpki/ca.cer); who operates the rpkid that host your children
-or publication clients is not strictly relevant to the authorization
-model, what matters is who holds the resources and is authorized to
-request and publish RPKI data derived from them.
+resource holder, but the software does not insist on this.
+
+[Er, well, rpkid doesn't, but myrpki now does -- "pure" hosting was an
+unused feature that fell by the wayside while simplifying the user
+interface.  It would be relatively straightforward to add it back if
+we ever need it for anything, but the mechanism it used to use no
+longer exists -- the old [myirbe] section of the config file has been
+collapsed into the [myrpki] section, so testing for existance of the
+[myrpki] section no longer works.  So we'll need an explicit
+configuration option, no big deal, just not worth chasing now.]
+
+A (perhaps) plausible use for this capability would be if you are an
+rpkid-running resource holder who wants for some reason to keep the
+resource-holding side of your operation completely separate from the
+rpkid-running side of your operation.  This is essentially the
+pure-hosting model, just with an internal hosted entity within a
+different part of your own organization.
 
 TROUBLESHOOTING
 
@@ -628,9 +399,9 @@ getting TLS errors, check to make sure that you're using all the right
 BPKI certificates and service contact URLs.
 
 TLS configuration errors are, unfortunately, notoriously difficult to
-debug, because connections due to misconfiguration usually fail early,
-deep in the guts of the OpenSSL TLS code, where there isn't enough
-application context available to provide useful error messages.
+debug, because connection failures due to misconfiguration happen
+early, deep in the guts of the OpenSSL TLS code, where there isn't
+enough application context available to provide useful error messages.
 
 If you've completed the steps above, everything appears to have gone
 OK, but nothing seems to be happening, the first thing to do is check