path: root/openssl/README

$Id$

OpenSSL hacked to add support for the RFC 3779 X.509 v3 extensions.



Some notes on OpenSSL internals.

O'Reilly "Network Security with OpenSSL" is a bit dated (four years
old, corresponds roughly to OpenSSL 0.9.7), but still appears to be
the closest thing there is to coherent documentation.  Some updates
and machine readable copies of examples are available at
http://www.opensslbook.com/.  In spite of its age, the book is useful
as it gives a readable overview of bit and pieces of OpenSSL's
internal programming environment which one would otherwise have to
absorb from the code via osmosis.  Chapter 10 is particularly useful,
as are the sections on error handling and abstract I/O (the ERR and
BIO packages, respectively) in chapter 4.

OpenSSL's own doc is very patchy, although fairly extensive in places.
Most of it eventually comes down to "Use the Source, Luke" with
pointers on which bit of source serves as an interesting example.

For x509v3 extentions, the place to start is doc/openssl.txt,
which, oddly, turns out to be mostly about certificate extensions.  It
gives an overview of the mechanisms, in particular of the method
routine interface for certificate extensions.  For something like the
RFC 3779 extensions, it's pretty clear that we need to use a "raw"
extension (which, as far as I can tell, just means that the RFC 3779
stuff is complicated enough that the extension handler has to do a lot
of work to deal with a complex ASN.1 structure that the rest of the
code doesn't know much about).

openssl.txt refers the reader to the certificate policy extension
handler as an example of how to write a raw extension. The relevant
file appears to be

    crypto/x509v3/v3_cpols.c

This is indeed a complicated mess, but it's a surprisingly good
tutorial, on about the tenth reading.  Note that this depends on
extension-specific typedefs defined in crypto/x509v3/x509v3.h.

Aside from the method interface and routines described in openssl.txt,
v3_cpols.c also includes a bunch of template junk for the ASN.1
encoder and decoder.  According to README.ASN1 (q.v.), the ASN.1 code
is in transition.  I haven't attempted to understand the "old" ASN.1
code; the new stuff is template driven.  README.ASN.1 mumbles about
this a bit, but the real doc on the template stuff appears to be the
definitions in crypto/asn1/asn1t.h.  Given that and a reading
knowledge of ASN.1, the template definitions in v3_cpols.c are fairly
obviously a direct hand translation from ASN.1 into C.

General note on global symbols in OpenSSL: always look at the header
file for any global symbol you're using.  In fact, it's probably best
to do a global search (m-x tags-search if you're an emacs user) for
all instances of a global symbol before attempting to use it, as there
are a lot of things that one just has to know about how all the global
stuff hangs together.  There are header files full of magic
definitions that one just has to one need to be extended.  There are
magic pre-sorted lists of handlers that one just needs to know about.
Little or none of this is documented.  Use the Source, Luke.

In some cases -portions- of files are automatically generated by Perl
scripts (eg, the per-type stack definitions in safestack.h).  Ouch.
If you see a large block of very repetitive stuff, check for comments
indicating that it's automatically generated.

Oh, and the indentation style is demented.

Header files you definitely need to read if you're going to touch this
stuff:

    crypto/x509v3/x509v3.h
    crypto/x509v3/ext_dat.h
    crypto/stack/safestack.h
    crypto/asn1/asn1t.h
    crypto/asn1/asn1.h
    crypto/objects/objects.h

Automatically generated header files you'll need to skim, then go read
the input files listed in the header comments and perhaps the
generating Perl code:

    crypto/objects/obj_mac.h
    crypto/objects/obj_dat.h

Nice example of "new" style ASN.1 (if it compiles, which the README
suggets it won't, sigh):

    demos/asn1/ocsp.c

Much of the code shows a heavy Perl influence, presumably dating back
clear to Eric Young.  Some of the internal data structure operators
have names that only make sense to a Perl programmer.  Where code is
automatically generated, it's done by Perl scripts.  The configuration
language for the whole package is a Perl script.  Assembly code is all
wrapped up inside perl scripts in a moderately clever attempt at being
able to write the assembly language only once and use it with various
assemblers with nontrivially different syntax.  Much of the
documentation markup (including manual pages) for the C code is .pod.

At this point I think I understand enough that I could write the RFC
3779 extension I/O routines, although no doubt some of it would be
take a while since I would be learning my way through some of the
library routines as I go.

I have not yet written up a syntax description for how we would encode
the RFC 3779 extensions in an openssl.conf file.  That should be
fairly straightforward, if a bit tedious.

I have not yet figured out where to hook in the extra goop that RFC
3779 will need for verification.  Making extensions critical is easy
enough, but the validation stuff in RFC 3779 2.3 and 3.3 needs to go
somewhere, I'm not quite sure where yet.  To the extent that I
understand how OpenSSL does such things (not very well, yet), the
"-purpose" flags to the CLI's x509 and verify commands (q.v.) look
like the right hooks.  I haven't (yet) investigated the code in back
of the "-purpose" mechanism.

At the moment the consensus appears to be that we should be using
OpenSSL 0.9.8b, in order to get SHA-256 support.  My research to date
has been on OpenSSL 0.9.8a, but I doubt that any of the stuff I've
been looking at has changed enough to matter.



This is a first cut at what the openssl.conf syntax might look like
for the RFC 3779 certificate extensions.  Syntax is admittedly
wretched, because it has to work with the existing OpenSSL code.
Within that restriction, I've attempted to make this look as much as
practical like the existing OpenSSL support for "multi-valued"
extensions.  RFC 3779 ASN.1 provided for easy reference.

Notes:

* Ranges are denoted with a hyphen, prefix lengths with a slash.
  I could tag ranges differently from the atomic types, but this
  seemed easier for the user to understand.

* The "@" syntax indicating indirection through a separate section
  is lifted from the stock OpenSSL multi-valued extension support.

* I didn't attempt to guess which addresses are IPv4 and which are
  IPv6 from the syntax, since the long form needed tags anyway.

* For the moment I'm assuming that the only addresses we really care
  about making (relatively) easy are unicast v4 and v6 addresses; the
  "afisafi" syntax is there for completeness, but I'm not currently
  worrying about making it easy to do, eg, multicast addresses.
  Adding support for other specific AFI/SAFI pairs should be
  straightforward.

* I'm particularly looking for feedback about things I've made
  impossible (syntax not general enough) or common things I've made
  harder than necessary.

###

   IPAddressFamily     ::= SEQUENCE {    -- AFI & optional SAFI --
      addressFamily        OCTET STRING (SIZE (2..3)),
      ipAddressChoice      IPAddressChoice }

   IPAddressChoice     ::= CHOICE {
      inherit              NULL, -- inherit from issuer --
      addressesOrRanges    SEQUENCE OF IPAddressOrRange }

   IPAddressOrRange    ::= CHOICE {
      addressPrefix        IPAddress,
      addressRange         IPAddressRange }

   IPAddressRange      ::= SEQUENCE {
      min                  IPAddress,
      max                  IPAddress }

   IPAddress           ::= BIT STRING


# An address extension all specified on one line

rfc3779-addr = critical, ipv4:10.1.1.1/32, ipv4:10.2.0.0-10.3.255.255

# An address extension all specified on one line, with inheritance

rfc3779-addr = critical, ipv4:inherit, ipv6:2002::/16

# An address extension using raw AFI/SAFI syntax.
#
# This doesn't really work, we can't do prefixes without knowing how long
# a particular kind of address is supposed to be.
#rfc3779-addr = critical, afisafi:000101:001122334455-aabbccddee

# An address extension using SAFIs

rfc3779-addr = critical, ipv4-safi:1:10.1.1.1/32, ipv6-safi:1:2002::/16

# Address extension using an indirect section

rfc3779-addr = critical, @addr-section

[addr-section]

ipv4.0 = 10.0.0.1
ipv4.1 = 10.0.1.0/24
ipv4.2 = 10.2.0.0 - 10.3.255.255
ipv6.0 = 2002:1::/64
ipv6.1 = 2002:2:: - 2002:8::ffff:ffff:ffff:ffff:ffff

###

   ASIdentifiers       ::= SEQUENCE {
       asnum               [0] EXPLICIT ASIdentifierChoice OPTIONAL,
       rdi                 [1] EXPLICIT ASIdentifierChoice OPTIONAL}

   ASIdentifierChoice  ::= CHOICE {
      inherit              NULL, -- inherit from issuer --
      asIdsOrRanges        SEQUENCE OF ASIdOrRange }

   ASIdOrRange         ::= CHOICE {
       id                  ASId,
       range               ASRange }

   ASRange             ::= SEQUENCE {
       min                 ASId,
       max                 ASId }

   ASId                ::= INTEGER

# ASID extension all specified on one line:

rfc3779-asid = critical, as:44, rdi:33-45

# ASID extension on one line using inheritance

rfc3779-asid = critical, as:55, rdi:inherit

# ASID extension using an indirect section

rfc3779-asid = critical, @asid-section

[asid-section]

as.0 = 44
as.1 = 55 - 77
rdi.0 = 33



Problem: how to do the top level ASN.1 template thingie for a SEQUENCE
(not SEQUENCE OF)?

Possible example: crypto/krb5/krb5_asn.c, KRB5_TKTBODY and KRB5_TICKET.

I -think- the flag combination I should be using is ASN1_TFLG_EXPLICIT
(context-specific explicit tags, which, according to mtr and my vague
recollection of ASN.1 tagging, is what RFC 3779 uses).

No, that's not right, kerberos tags the SEQUENCE itself, which we
don't.  So either leave the flags field empty or use
ASN1_TFLG_IMPLICIT.

But this is all very spooky, I can't find a good example, and it seems
wrong somehow that I'm ending up with template that points to a
sequence instead of being a sequence.

Hmm, maybe that means I need to call lower-level macros that do
specific parts of the sequence constructor, then build a template with
the SEQUENCE flags?

Well, try what we have and see if it works, then figure out why not.

Ok, one more try: crypto/x509v3/v3_{pci,pcia}.c implement RFC 3820 and
show a SEQUENCE used as the top level template.  Try it that way.

Hmm, maybe openssl.txt really meant v3_{pci,pcia}.c as an example?



Oooh, crypto/stack/ includes stack sorting functions.  With the right
structure definition and comparision function, this might solve a
whole mess of our canonicalization problems.  General plan:

1) Define an internal stack type for each of the extensions, with
   fields organized to allow us to handle everything with a single
   sort operation.  Eg, major key is afi/safi, expand prefixes to
   ranges or some such, etc.

2) Set an appropiate comparison function (see sk_set_cmp_func())

3) Parse config data into internal stack.

4) Call sk_sort().

5) Loop calling sk_shift() (or sk_pop()) to pull items off the sorted
   stack so we can insert them into our real structure in the right
   canonical form.

Hmm.  What about other ways of building up the real structure?  Would
it be better to have a "canoicalize structure" function for each
extension type?  What about allowing a caller to say "no thanks it's
already in canonical form please leave it alone"?

util/mkstack.pl finds DECLARE_STACK_OF() declarations and generates
safestack definitions automatically.  Be afraid.  Be very very afraid.



Upon closer examination of RFC 3779, it's not possible to do a
completely general handler for unknown AFIs, because it's the AFI that
tells one that maximum length of an address (which in turn is required
for certain operations, eg, canonization).

So I guess we end up with:

a) IPv4:foo, IPv4-SAFI:foo, IPv6:foo, IPv6-SAFI:foo (or something like
   that -- perhaps encode SAFI names too, eg IPv4-Unicast:foo), and

b) A table of known AFIs telling us the length of an address for that
   AFI.  For the moment I'm probably not going to bother filling that
   in for anything but IPv4 and IPv6, but having the mechanism in
   place for other AFIs seems harmless.



Current plan for addr_canonize(): sort function for outer sequence
should be straightforward.  One could in theory write a sort function
for the inner sequence, but doing so would require expanding addresses
both in the sort function and in the subsequent cleanup loop in
addr_canonize().  It would probably be simpler (and faster) to use a
temporary stack containing pointers to the IPAddressOrRange elements
and expanded versions of the addresses.

Hmm, the alternative would be to write a comparision routine which
expands a pair of bitstrings as addresses, and write the rest of the
code in terms of that.  Might be a useful function to have anyway.

int addr_cmp(int afi, ASN1_BIT_STRING *a, unsigned char fill_a,
                      ASN1_BIT_STRING *b, unsigned char fill_b);

Problem is, this requires data not available in a sk_sort() handler.
So I think we're stuck with the temporary stack approach. 

No, we can use a closure.



addr_canonize() code is not right yet, mostly because merging prefixes
and ranges is tricky.  Logic that checks whether they need to be
merged is probably ok, it's the code that does the merge that's whacked.

Perhaps what we need is a function that takes two prefix-or-ranges
(which the caller asserts should be merged), and generates the prefix
or range that starts with the first one and ends with the second.
Would probably be easier than what the code does now.  Worry about
optimizing this later, if it matters, get it right first.

Current code in addr_canonize() is much more complicated than
necessary, needs reorganization.  All we really need to do is:

1) Expand begin and end addresses of A and B.

2) Check whether they overlap (or are adjacent).

3) If so, replace them with a merged prefix or range.



Doh.  A lot of the missing documentation is buried in ssleay.txt,
which the other documentation says not to read because it's so old.
But it's where Eric explains all the basic data structures and
expected usage as of the dawn of time, so most of the stuff that's so
old that it's undocumented is really documented there.  Sigh.

Anyway, it looks like the xxx_new() functions already set pointers of
sub structures to NULL or allocate the substructures (have to check
which for the things we care about, I guess), and the xxx_free()
functions already clean up complex structures.  So be sure to set
unused pointers to NULL if one has been fiddling.

foo->type fields may be initialized to -1 too, need to check that.

So the destructor routines I wrote are unnecessary, and some of the
memset()s may be actively harmful.  Need to clean that up.

Also need to check whether we need to call sk_xxx_pop_free() manually
during cleanup.  Seems unlikely, the template driven code has to be
able to handle that one would think.

Make sure that memory leak detection (MDEBUG in very old ssleay,
figure out what it is now) is turned on.



"make update" in the top level runs all the magic perl code that
grovels through the code generating error codes, safe stacks, etc.

My initial test configuration:

./Configure debug -DDEBUG_SAFESTACK

This seems to be attempting to pull in devel/ElectricFence (-lefence),
so I installed that.

[Later...]

Ok, the default debugging options are driving me nuts.  ElectricFence
is very slow, although no doubt I might want it anyway at some point.
But it's the BN debugging printouts that finally pushed me into adding
a "debug-sra" config of my own for the options I want.



The identifiers we need are already in crypto/objects/objects.txt:

id-pe 7			: sbgp-ipAddrBlock
id-pe 8			: sbgp-autonomousSysNum

Now I just have to figure out how to use them. :)

crypto/objects/objects.h appears to be hand-maintained, not entirely
surprising given that it appears to be the master NID assignment
table.

Oh good, if USE_OBJ_MAC is defined objects.h sucks in obj_mac.h which
is automatically generated from objects.txt, so we're already there.
NIDs have already been assigned and everything, just have to use them.



-Werror is set in compilation flags, so can't use #warning.

Remember to clean up the new stuff ni x509v3.h!




## [.gdbinit]

# this cert generates a core dump, might be interesting to find out why
set arg x509 -noout -text -in /u/sra/isc/route-pki/ftp.apnic.net/pub/test-certs/fc0000000003.cert

# but maybe let's start with one which is just broken rather than dumping core
set arg x509 -noout -text -in /u/sra/isc/route-pki/ftp.apnic.net/pub/test-certs/fc4884ce32f6.cert

# hmm, that one dumps core too, what fun

break i2r_IPAddrBlocks


##

The decoded ASN.1 structures are not what the code expects.  Hmm.  The
AS number stuff is not dumping core, have not yet figured out whether
it's really working or is just not noticing that it's broken.

i2r_IPAddressOrRanges (out=0x2b5b6fc0, indent=18, aors=0x2b9a0ff0, afi=1) at v3_addr.c:152
(gdb) p aors
$1 = (const IPAddressOrRanges *) 0x2b9a0ff0
(gdb) p *aors
Error accessing memory address 0x2b9a0ff0: Bad address.
(gdb) n
(gdb) n

Program received signal SIGSEGV, Segmentation fault.
0x080fba4b in sk_value (st=0x2b9a0ff0, i=0) at stack.c:310

The layer above this looks right (appears to be a valid stack, the
addressFamily field looks plausible).

[Later...]

Hmm, is SEQUENCE(CHOICE(foo,bar)) a problem?  ASID code works, ADDR
does not, place where ADDR breaks is a SEQUENCE OF CHOICE.

Yep, that did it.  Ok, display (i2r) code works.  Still need to debug
v2i code.

i2r code needs minor work, needs to break long lines of ASIDs.
Deal with this later.

Punctuation discrepancies between ASID and ADDR code, ASID does

  "foo - bar"

for ranges while ADDR does

  "foo-bar"

ASID is probably right here, more readable.  Should probably add the
strspn() hackery to ASID code so it can tolerate looser input.

I've seen occasional printouts of what look like empty address
extensions, need to check that those really are empty (as opposed to
just a printout bug that's aborting i2r...).



asid v2i is losing on ranges because the ASN1_INTEGER reader code
doesn't like our punctuation.  Simplest answer would be to use
strtol() to read the number then convert it with ASN1_INTEGER_set(),
which would work fine for now but won't work with 32-bit AS numbers
(ASN1_INTEGER_set() wants long, not unsigned long).

Less efficient answer would be to strdup() the substring we want, etc.

Scary efficient answer would be to clone ASN1_INTEGER_set() for
unsigned long.

While efficiency is tempting, it's probably misguided in this case.
At the moment we are just dealing with ASN.1 integers at a fairly
abstract level and don't care a whit about machine representation,
which is probably the right thing since the user only cares about
ASN.1 and text.  So strdup() is probably the right answer.

addr v2i is having similar problems, inet_pton() doesn't like the
slash of a prefix.  Same answer I guess.

Insertion algorithm for IPAddrBlocks doesn't appear to be working
right, I'm getting duplicate IPAddressFamily elements, each containing
a single address.

@foo section indirection isn't working quite right yet: the conf
library does the right thing, but the CONF_VALUE->name we get back has
the numeric tag ("as.0", "as.1", etc) which we're not handling.  Check
what other extensions (eg subjectAltName) do about this.

Ah, we should use name_cmp() instead of strcmp(), ok.

Might be able to get rid of inet_pton(): see ipv4_from_asc() and
ipv6_from_asc() in v3_utl.c.

Ok, name_cmp() nailed the indirect section problem, and the
IPAddrBlocks insertion problem was a bad pointer, fixed.  Amazingly
enough, the addr code to collapse ranges into prefixes is working.

ASID code is not merging individual AS numbers into ranges properly.

Time for breakfast.