aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorRob Austein <sra@hactrn.net>2019-04-15 16:40:39 +0000
committerRob Austein <sra@hactrn.net>2019-04-15 16:40:39 +0000
commit5decc49d4672d29c6b91fc01825b9dbef074b00a (patch)
tree46c400f99b939c0c95764f1b5f7851ba177857e5
parentb0f2c8c1c607648b11a5499a425bc87c0bd6fa22 (diff)
Add missing README and copyright
-rw-r--r--README.md99
-rwxr-xr-xbaiji14
2 files changed, 113 insertions, 0 deletions
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..be3692a
--- /dev/null
+++ b/README.md
@@ -0,0 +1,99 @@
+baiji
+=====
+
+Baiji is a package building tool for Debian environments. The overall
+design borrows heavily from an existing tool called `whalebuilder`,
+which in turn is in effect a reimplementation of `pbuilder` using
+Docker to do the heavy lifting.
+
+Clean room build environment
+----------------------------
+
+`pbuilder` is one of several tools all of which have the same basic
+goal: building packages in an environment specified by the package's
+`control` file with minimal dependencies on the native environment of
+the machine where the build is being run.
+
+`pbuilder` is probably the oldest surviving example of this kind of
+build environment for the Debian universe, and uses a fairly simple
+approach: it keeps a pristine build environment in a compressed
+tarball, which it unpacks into a freshly-chrooted tree for every
+build, thus insuring a predictable environment. The pristine base
+environment only includes the "build-essentials" environment that
+every Debian package is allowed to assume is always installed:
+anything else specified in the `control` file is installed on demand
+every time. While a bit finicky to set up, this is pretty simple in
+overall concept, and pretty reliable. Unfortunately, it's also pretty
+slow.
+
+`pbuilder` lacks direct support for dependencies on packages not known
+to the Debian project, but it does support hook scripts, which is how
+we taught it to use the local package collections that our current
+build system stores in `/var/cache/pbuilder/local-packages/`.
+
+If one looks closely at this setup, it turns out to have a lot in
+common with Docker containers. The specific mechanisms are different,
+but the overall effect is the same: an isolated environment running a
+controlled set of packages and scripts independent of the machine on
+which it's running. The difference is that Docker has a lot of
+support for building and distributing images based on other images.
+
+There's a lovely little tool called `whalebuilder` which to a first
+approximation is a reimplementation of `pbuilder` on top of Docker.
+It turns out that `whalebuilder` doesn't do quite what we want, but
+it's close enough to serve as a (very) useful model for a new tool
+
+baiji
+-----
+
+`baiji` is intended to incorporate the most useful features of both
+`whalebuilder` and `git-buildpackage`, along with some custom features
+such as the ability to load locally-built `.deb` packages directly
+into the constructed Docker image.
+
+Like both `pbuilder` and `whalebuilder`, `baiji` uses a "base image",
+which is constructed using the `debbootstrap` package, along with a
+few things needed by `baiji` itself.
+
+Like `whalebuilder` but unlike `pbuilder`, `baiji` also builds
+package-specific images for building a particular source package.
+While this may sound wasteful, it turns out that it's not: everything
+involved in the per-package image construction process is work that
+`baiji` would have to do anyway, and by saving the result `baiji`
+makes it possible to reuse that image if it turns out that one has to
+build the same version of the same source package repeatedly. By
+default, `baiji` only generates a new package-specific build image
+when the package's version or build dependencies have changed, but the
+user can override this to force a new image.
+
+Also like `whalebuilder` but unlike `pbuilder`, `baiji` performs the
+build itself as a non-root user, which is a better fit for the base
+Debian model of building packages, and runs with networking turned
+off, so that it can catch undeclared dependencies like a package
+quietly attempting to `pip install` 25 new packages from PyPi.
+
+Like `debuild` and the `pdebuild` but unlike `whalebuilder`, `baiji`
+can generate the source package from the source tree when needed.
+
+As mentioned above, `baiji` supports pre-loading local binary packages
+directly into a local APT repository within the package-specific
+Docker image.
+
+`baiji`'s base images can be pre-configured to know about local APT
+servers,, so that any packages needed for a build can be downloaded
+directly. `apt` running within the usual build tools will use the
+package version numbers to decide which version of a package to use
+when more than one is available.
+
+`baiji`'s current Interface to Docker just launches the Docker CLI
+tool in a subprocess, which turns out to be a lot simpler than using
+the official Docker Python API; the latter is only available from
+GitHub and PyPi, and depends on enough other PyPi packages that it's a
+bit of a maintenance nightmare. Perhaps this will calm down someday,
+but for now it's worth the minor inefficiency of using the Docker CLI
+to avoid the swamp.
+
+Unlike `whalebuilder`, which uses uses Ruby's built-in Jinja-like
+template mechanism, `baiji` currently just includes the handful of
+templates it needs inline using Python's `str.format()` method. Some
+day `baiji` may outgrow this, but it suffices for now.
diff --git a/baiji b/baiji
index 23d0696..f39d051 100755
--- a/baiji
+++ b/baiji
@@ -1,5 +1,19 @@
#!/usr/bin/env python
+# Copyright (c) 2018-2019, Grunchweather Associates
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+# REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+# LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+# OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+# PERFORMANCE OF THIS SOFTWARE.
+
import debian.changelog
import debian.deb822
import subprocess
6 22:07:15 +0000 committer Rob Austein <sra@hactrn.net> 2010-04-16 22:07:15 +0000 Use html2text instead of lynx for dumping doc to flat text' href='/sra/rpki.net/commit/rpkid/doc/Publication?h=zone-cleanup&id=27f1137ac1e65d5bc1e21d43c9bdf6166dce6bb0'>27f1137a
94bad6e5
27f1137a








94bad6e5
27f1137a


94bad6e5
27f1137a


c495c088
27f1137a

























94bad6e5
27f1137a



27f1137a




27f1137a








































1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
                 









                              
                    
 

                                                                        
 


                                                                           
 

                                                                               
                                                                          


                                                                              
 


                                                                               
 
                                
 



                                                                            
 
                 








                                                                            







                                                                              
 
                 










                                                                               



















                                                                               
 
                        















                                                                               
                      



                                                                           
              


                                                                      
                   








                                                                               
              


                                                                
               


                                                 
                                                                             

























                                                                               
                                          



                                                                             




                                                                           








































                                                                               
RPKI Engine � 1.0

* Main�_Page
* Related�_Pages
* Packages
* Classes
* Files
*  [Search              ]

* RPKI_Engine_Reference_Manual

Publication protocol

The publication protocol is really two separate client/server protocols,
between different parties.

The first is a configuration protocol for an IRBE to use to configure a
publication engine, the second is the interface by which authorized clients
request publication of specific objects.

Much of the architecture of the publication protocol is borrowed from the left-
right_protocol: like the left-right protocol, the publication protocol uses
CMS-wrapped XML over HTTP with the same eContentType OID and the same HTTP
content-type, and the overall style of the XML messages is very similar to the
left-right protocol. All operations allow an optional "tag" attribute to allow
batching.

The publication engine operates a single HTTP server which serves both of these
subprotocols. The two subprotocols share a single server port, but use distinct
URLs to allow demultiplexing.

 Publication control subprotocol

The control subprotocol reuses the message-passing design of the left-right
protocol. Configured objects support the "create", "set", "get", "list", and
"destroy" actions, or a subset thereof when the full set of actions doesn't
make sense.

 <config/> object

The <config/> object allows configuration of data that apply to the entire
publication server rather than a particular client.

There is exactly one <config/> object in the publication server, and it only
supports the "set" and "get" actions -- it cannot be created or destroyed.

Payload data which can be configured in a <config/> object:

  bpki_crl (element):
      This is the BPKI CRL used by the publication server when signing the CMS
      wrapper on responses in the publication subprotocol. As the CRL must be
      updated at regular intervals, it's not practical to restart the
      publication server when the BPKI CRL needs to be updated. The BPKI model
      doesn't require use of a BPKI CRL between the IRBE and the publication
      server, so we can use the publication control subprotocol to update the
      BPKI CRL.

 <client/> object

The <client/> object represents one client authorized to use the publication
server.

The <client/> object supports the full set of "create", "set", "get", "list",
and "destroy" actions. Each client has a "client_handle" attribute, which is
used in responses and must be specified in "create", "set", "get", or "destroy"
actions.

Payload data which can be configured in a <client/> object:

  base_uri (attribute):
      This is the base URI below which this client is allowed to publish data.
      The publication server may impose additional constraints in the case of a
      child publishing beneath its parent.

  bpki_cert (element):
      BPKI CA certificate for this <client/>. This is used as part of the
      certificate chain when validating incoming TLS and CMS messages. If the
      bpki_glue certificate is in use (below), the bpki_cert certificate should
      be issued by the bpki_glue certificate; otherwise, the bpki_cert
      certificate should be issued by the publication engine's bpki_ta
      certificate.

  bpki_glue (element):
      Another BPKI CA certificate for this <client/>, usually not needed.
      Certain pathological cross-certification cases require a two-certificate
      chain due to issuer name conflicts. If used, the bpki_glue certificate
      should be the issuer of the bpki_cert certificate and should be issued by
      the publication engine's bpki_ta certificate; if not needed, the
      bpki_glue certificate should be left unset.

 Publication subprotocol

The publication subprotocol is structured somewhat differently from the
publication control protocol. Objects in the publication subprotocol represent
objects to be published or objects to be withdrawn from publication. Each kind
of object supports two actions: "publish" and "withdraw". In each case the XML
element representing hte object to be published or withdrawn has a "uri"
attribute which contains the publication URI. For "publish" actions, the XML
element body contains the DER object to be published, encoded in Base64; for
"withdraw" actions, the XML element body is empty.

In theory, the detailed access control for each kind of object might be
different. In practice, as of this writing, access control for all objects is a
simple check that the client's "base_uri" is a leading substring of the
publication URI. Details of why access control might need to become more
complicated are discussed in a later section.

 <certificate/> object

The <certificate/> object represents an RPKI certificate to be published or
withdrawn.

 <crl/> object

The <crl/> object represents an RPKI CRL to be published or withdrawn.

 <manifest/> object

The <manifest/> object represents an RPKI publication manifest to be published
or withdrawn.

Note that part of the reason for the batching support in the publication
protocol is because every publication or withdrawal action requires a new
manifest, thus every publication or withdrawal action will involve at least two
objects.

 <roa/> object

The <roa/> object represents a ROA to be published or withdrawn.

 Error handling

Error in this protocol are handled at two levels.

Since all messages in this protocol are conveyed over HTTP connections, basic
errors are indicated via the HTTP response code. 4xx and 5xx responses indicate
that something bad happened. Errors that make it impossible to decode a query
or encode a response are handled in this way.

Where possible, errors will result in a <report_error/> message which takes the
place of the expected protocol response message. <report_error/> messages are
CMS-signed XML messages like the rest of this protocol, and thus can be
archived to provide an audit trail.

<report_error/> messages only appear in replies, never in queries. The
<report_error/> message can appear in both the control and publication
subprotocols.

The <report_error/> message includes an optional "tag" attribute to assist in
matching the error with a particular query when using batching.

The error itself is conveyed in the error_code (attribute). The value of this
attribute is a token indicating the specific error that occurred. At present
this will be the name of a Python exception; the production version of this
protocol will nail down the allowed error tokens here, probably in the RelaxNG
schema.

The body of the <report_error/> element itself is an optional text string; if
present, this is debugging information. At present this capabilty is not used,
debugging information goes to syslog.

 Additional access control considerations.

As detailed above, the publication protocol is trivially simple. This glosses
over two bits of potential complexity:

* In the case where parent and child are sharing a repository, we'd like to
  nest child under parent, because testing has demonstrated that even on
  relatively slow hardware the delays involved in setting up separate rsync
  connections tend to dominate synchronization time for relying parties.

* The repository operator might also want to do some checks to assure itself
  that what it's about to allow the RPKI engine to publish is not dangerous
  toxic waste.

The up-down protocol includes a mechanism by which a parent can suggest a
publication URI to each of its children. The children are not required to
accept this hint, and the children must make separate arrangements with the
repository operator (who might or might not be the same as the entity that
hosts the children's RPKI engine operations) to use the suggested publication
point, but if everything works out, this allows children to nest cleanly under
their parents publication points, which helps reduce synchronization time for
relying parties.

In this case, one could argue that the publication server is responsible for
preventing one of its clients (the child in the above description) from
stomping on data published by another of its clients (the parent in the above
description). This goes beyond the basic access check and requires the
publication server to determine whether the parent has given its consent for
the child to publish under the parent. Since the RPKI certificate profile
requires the child's publication point to be indicated in an SIA extension in a
certificate issued by the parent to the child, the publication engine can infer
this permission from the parent's issuance of a certificate to the child.
Since, by definition, the parent also uses this publication server, this is an
easy check, as the publication server should already have the parent's
certificate available by the time it needs to check the child's certificate.

The previous paragraph only covers a "publish" action for a <certificate/
> object. For "publish" actions on other objects, the publication server would
need to trace permission back to the certificate issued by the parent; for
"withdraw" actions, the publication server would have to perform the same
checks it would perform for a "publish" action, using the current published
data before withdrawing it. The latter in turn implies an ordering constraint
on "withdraw" actions in order to preserve the data necessary for these access
control decisions; as this may prove impractical, the publication server may
probably need to make periodic sweeps over its published data looking for
orphaned objects, but that's probably a good idea anyway.

Note that, in this publication model, any agreement that the repository makes
to publish the RPKI engine's output is conditional upon the object to be
published passing whatever access control checks the publication server
imposes.