Whack rcynicng to produce an XML file which rcynic-html can parse:

needs further work, but produces somewhat-useful output now.

Start merging command line argument (argparse) and config file
(ConfigParser) support into a unified library module, in an attempt to
reduce the amount of code duplication and reign in some of the
configuration chaos.  Trigger for this is all of the RP programs now
needing SQL configuration.

Whack rpki.rcynicdb.models to work with PostgreSQL, which has length
restrictions on indexed columns.

svn path=/branches/tk705/; revision=6239

2 files changed, 73 insertions, 3 deletions

diff --git a/rpki/config.py b/rpki/config.py
index a9bd3219..3cc16626 100644
--- a/rpki/config.py
+++ b/rpki/config.py
@@ -23,6 +23,7 @@ ConfigParser module.
 """
 
 import ConfigParser
+import argparse
 import logging
 import os
 import re
@@ -272,3 +273,60 @@ class parser(object):
             rpki.up_down.content_type = self.get("up_down_content_type")
         except ConfigParser.NoOptionError:
             pass
+
+
+def argparser(section = None, doc = None, cfg_optional = False):
+    """
+    First cut at a combined configuration mechanism based on ConfigParser and argparse.
+
+    General idea here is to do an initial pass on the arguments to handle the config file,
+    then return the config file and a parser to use for the rest of the arguments.
+    """
+
+    # Basic approach here is a variation on:
+    # http://blog.vwelch.com/2011/04/combining-configparser-and-argparse.html
+
+    # For most of our uses of argparse, this should be a trivial
+    # drop-in, and should reduce the amount of repetitive code.  There
+    # are a couple of special cases which will require attention:
+    #
+    # - rpki.rtr: These modules have their own handling of all the
+    #   logging setup, and use an argparse subparser.  I -think- that
+    #   the way they're already handling the logging setup should work
+    #   fine, but there may be a few tricky bits reconciling this code
+    #   with the more generalized version in rpki.log.
+    #
+    # - rpki.rpkic: Use of argparse here is very complicated due to
+    #   support for both the external command line and the internal
+    #   command loop.  Overall it works quite well, but the setup is
+    #   tricky.  rpki.rpkic.main.top_argparse may need to move outside
+    #   the main class, but that may raise its own issues.  Maybe we
+    #   can get away with just replacing the current setup of
+    #   top_argparser with a call to this function and otherwise
+    #   leaving the whole structure alone?  Try and see, I guess.
+
+    # Setting cfg_optional here doesn't really work, because the cfg
+    # object returned here is separate from the one that the Django
+    # ORM gets when it tries to look for databases.  Given that just
+    # about everything which uses this module also uses Django,
+    # perhaps we should just resign ourselves to the config being a
+    # global thing we read exactly once, so we can stop playing this
+    # game.
+
+    topparser = argparse.ArgumentParser(add_help = False)
+    topparser.add_argument("-c", "--config",
+                           default = os.getenv(rpki_conf_envname, default_filename),
+                           help = "override default location of configuration file")
+
+    cfgparser = argparse.ArgumentParser(parents = [topparser], add_help = False)
+    cfgparser.add_argument("-h", "--help", action = "store_true")
+
+    args, remaining_argv = cfgparser.parse_known_args()
+
+    cfg = parser(section       = section,
+                 set_filename  = args.config,
+                 allow_missing = cfg_optional or args.help)
+
+    argparser = argparse.ArgumentParser(parents = [topparser], description = doc)
+
+    return cfg, argparser
diff --git a/rpki/rcynicdb/models.py b/rpki/rcynicdb/models.py
index 09d513d5..185482b1 100644
--- a/rpki/rcynicdb/models.py
+++ b/rpki/rcynicdb/models.py
@@ -31,14 +31,26 @@ class RRDPSnapshot(models.Model):
 #
 # https://docs.djangoproject.com/en/1.9/ref/models/fields/#django.db.models.ForeignKey.on_delete
 #
-# Might also want to provide names for the reverse relationships, code uses blah_set for now.
+# Might also want to provide names for the reverse relationships, code
+# uses blah_set for now.
+
+# Setting unique = True on the der field breaks with PostgreSQL, see
+# https://code.djangoproject.com/ticket/14904
+#
+# In theory collisions on sha256 are possible, but in practice they're
+# not going to occur by accident.  Setting unique = True on the sha256
+# field risks deliberate collisions, defending against that would
+# require detecting the collision and figuring out which is the
+# attacking object (easy in theory, as it probably won't validate),
+# then figuring out what to do about it (possibly harder -- do we drop
+# an entire RRDP zone because of one evil object?).
 
 class RPKIObject(models.Model):
-    der           = models.BinaryField(unique = True)
+    der           = models.BinaryField() # unique = True
     uri           = models.TextField()
     aki           = models.SlugField(max_length = 40)  # hex SHA-1
     ski           = models.SlugField(max_length = 40)  # hex SHA-1
-    sha256        = models.SlugField(max_length = 64) # hex SHA-256
+    sha256        = models.SlugField(max_length = 64, unique = True) # hex SHA-256
     retrieved     = models.ForeignKey(Retrieval)
     authenticated = models.ManyToManyField(Authenticated)
     snapshot      = models.ManyToManyField(RRDPSnapshot)