packaging/ostree
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Rewrite manual in mkdocs

Browse Source 
I don't much like Docbook (and am considering converting the man pages
too), but let's start with the manual.

I looked at various documentation generators (there are a lot), and
I had a few requirements:

 - Markdown
 - Packaged in Fedora
 - Suitable for upload to a static webserver

`mkdocs` seems to fit the bill.
This commit is contained in:
Colin Walters committed 2016-01-28 09:31:37 -05:00
1 parent 32c360b5a0
commit 64ebe2b82a
17 files changed
+692 -1023
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/manual/adapting-existing.md
+159
View File
@@ -0,0 +1,159 @@
# Adapting existing mainstream distributions
## System layout
First, OSTree encourages systems to implement
[UsrMove](http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/)
This is simply to avoid the need for more bind mounts. By default
OSTree's dracut hook creates a read-only bind mount over `/usr`; you
can of course generate individual bind-mounts for `/bin`, all the
`/lib` variants, etc. So it is not intended to be a hard requirement.
Remember, because by default the system is booted into a `chroot`
equivalent, there has to be some way to refer to the actual physical
root filesystem. Therefore, your operating system tree should contain
an empty `/sysroot` directory; at boot time, OSTree will make this a
bind mount to the physical / root directory. There is precedent for
this name in the initramfs context. You should furthermore make a
toplevel symbolic link `/ostree` which points to `/sysroot/ostree`, so
that the OSTree tool at runtime can consistently find the system data
regardless of whether it's operating on a physical root or inside a
deployment.
Because OSTree only preserves `/var` across upgrades (each
deployment's chroot directory will be garbage collected
eventually), you will need to choose how to handle other
toplevel writable directories specified by the [Filesystem Hierarchy Standard](http://www.pathname.com/fhs/")
Your operating system may of course choose
not to support some of these such as `/usr/local`, but following is the
recommended set:
- `/home` → `/var/home`
- `/opt` → `/var/opt`
- `/srv` → `/var/srv`
- `/root` → `/var/roothome`
- `/usr/local` → `/var/local`
- `/mnt` → `/var/mnt`
- `/tmp` → `/sysroot/tmp`
Furthermore, since `/var` is empty by default, your operating system
will need to dynamically create the <emphasis>targets</emphasis> of
these at boot. A good way to do this is using `systemd-tmpfiles`, if
your OS uses systemd. For example:
```
d /var/log/journal 0755 root root -
L /var/home - - - - ../sysroot/home
d /var/opt 0755 root root -
d /var/srv 0755 root root -
d /var/roothome 0700 root root -
d /var/usrlocal 0755 root root -
d /var/usrlocal/bin 0755 root root -
d /var/usrlocal/etc 0755 root root -
d /var/usrlocal/games 0755 root root -
d /var/usrlocal/include 0755 root root -
d /var/usrlocal/lib 0755 root root -
d /var/usrlocal/man 0755 root root -
d /var/usrlocal/sbin 0755 root root -
d /var/usrlocal/share 0755 root root -
d /var/usrlocal/src 0755 root root -
d /var/mnt 0755 root root -
d /run/media 0755 root root -
```
Particularly note here the double indirection of `/home`. By default,
each deployment will share the global toplevel `/home` directory on
the physical root filesystem. It is then up to higher levels of
management tools to keep <filename>/etc/passwd</filename> or
equivalent synchronized between operating systems. Each deployment
can easily be reconfigured to have its own home directory set simply
by making `/var/home` a real directory. </chapter>
## Booting and initramfs technology
OSTree comes with optional dracut+systemd integration code that parses
the `ostree=` kernel command line argument in the initramfs, and then
sets up the read-only bind mount on `/usr`, a bind mount on the
deployment's `/sysroot` to the physical `/`, and then finally uses
`mount(MS_MOVE)` to make the deployment root appear to be the root
filesystem before telling systemd to switch root.
If you are not using dracut or systemd, using OSTree should still be
possible, but you will have to write the integration code. Patches to
support other initramfs technologies and init systems, if sufficiently
clean, will likely be accepted upstream.
A further specific note regarding `sysvinit`: OSTree used to support
recording device files such the `/dev/initctl` FIFO, but no longer
does. It's recommended to just patch your initramfs to create this at
boot.
## /usr/lib/passwd
Unlike traditional package systems, OSTree trees contain *numeric* uid
and gids. Furthermore, it does not have a `%post` type mechanism
where `useradd` could be invoked. In order to ship an OS that
contains both system users and users dynamically created on client
machines, you will need to choose a solution for `/etc/passwd`. The
core problem is that if you add a user to the system for a daemon, the
OSTree upgrade process for `/etc` will simply notice that because
`/etc/passwd` differs from the previous default, it will keep the
modified config file, and your new OS user will not be visible. The
solution chosen for the [Gnome Continuous](https://live.gnome.org/Projects/GnomeContinuous) operating
system is to create `/usr/lib/passwd`, and to include a NSS module
[nss-altfiles](https://github.com/aperezdc/nss-altfiles) which
instructs glibc to read from it. Then, the build system places all
system users there, freeing up `/etc/passwd` to be purely a database
of local users. See also a more recent effort from [Systemd Stateless](http://0pointer.de/blog/projects/stateless.html)
## Adapting existing package managers
The largest endeavor is likely to be redesigning your distribution's
package manager to be on top of OSTree, particularly if you want to
keep compatibility with the "old way" of installing into the physical
`/`. This section will use examples from both `dpkg` and `rpm` as the
author has familiarity with both; but the abstract concepts should
apply to most traditional package managers.
There are many levels of possible integration; initially, we will
describe the most naive implementation which is the simplest but also
the least efficient. We will assume here that the admin is booted
into an OSTree-enabled system, and wants to add a set of packages.
Many package managers store their state in `/var`; but since in the
OSTree model that directory is shared between independent versions,
the package database must first be found in the per-deployment `/usr`
directory. It becomes read-only; remember, all upgrades involve
constructing a new filesystem tree, so your package manager will also
need to create a copy of its database. Most likely, if you want to
continue supporting non-OSTree deployments, simply have your package
manager fall back to the legacy `/var` location if the one in `/usr`
is not found.
To install a set of new packages (without removing any existing ones),
enumerate the set of packages in the currently booted deployment, and
perform dependency resolution to compute the complete set of new
packages. Download and unpack these new packages to a temporary
directory.
Now, because we are merely installing new packages and not
removing anything, we can make the major optimization of reusing
our existing filesystem tree, and merely
*layering* the composed filesystem tree of
these new packages on top. A command like this:
```
ostree commit -b osname/releasename/description
--tree=ref=$osname/releasename/description
--tree=dir=/var/tmp/newpackages.13A8D0/
```
will create a new commit in the `$osname/releasename/description`
branch. The OSTree SHA256 checksum of all the files in
`/var/tmp/newpackages.13A8D0/` will be computed, but we will not
re-checksum the present existing tree. In this layering model,
earlier directories will take precedence, but files in later layers
will silently override earlier layers.
Then to actually deploy this tree for the next boot:
`ostree admin deploy <replaceable>osname/releasename/description`
docs/manual/atomic-upgrades.md
+116
View File
@@ -0,0 +1,116 @@
# Atomic Upgrades
## You can turn off the power anytime you want...
OSTree is designed to implement fully atomic and safe upgrades;
more generally, atomic transitions between lists of bootable
deployments. If the system crashes or you pull the power, you
will have either the old system, or the new one.
## Simple upgrades via HTTP
First, the most basic model OSTree supports is one where it replicates
pre-generated filesystem trees from a server over HTTP, tracking
exactly one ref, which is stored in the `.origin` file for the
deployment. The command `ostree admin upgrade`
implements this.
o begin a simple upgrade, OSTree fetches the contents of the ref from
the remote server. Suppose we're tracking a ref named
`exampleos/buildmaster/x86_64-runtime`. OSTree fetches the URL
`http://$example.com/repo/refs/exampleos/buildmaster/x86_64-runtime`,
which contains a SHA256 checksum. This determines the tree to deploy,
and `/etc` will be merged from currently booted tree.
If we do not have this commit, then, then we perform a pull process.
At present (without static deltas), this involves quite simply just
fetching each individual object that we do not have, asynchronously.
Put in other words, we only download changed files (zlib-compressed).
Each object has its checksum validated and is stored in `/ostree/repo/objects/`.
Once the pull is complete, we have all the objects locally
we need to perform a deployment.
## Upgrades via external tools (e.g. package managers)
As mentioned in the introduction, OSTree is also designed to allow a
model where filesystem trees are computed on the client. It is
completely agnostic as to how those trees are generated; hey could be
computed with traditional packages, packages with post-deployment
scripts on top, or built by developers directly from revision control
locally, etc.
At a practical level, most package managers today (`dpkg` and `rpm`)
operate "live" on the currently booted filesystem. The way they could
work with OSTree is instead to take the list of installed packages in
the currently booted tree, and compute a new filesystem from that. A
later chapter describes in more details how this could work:
[adapting-existing.md](Adapting Existing Systems).
For the purposes of this section, let's assume that we have a
newly generated filesystem tree stored in the repo (which shares
storage with the existing booted tree). We can then move on to
checking it back out of the repo into a deployment.
## Assembling a new deployment directory
Given a commit to deploy, OSTree first allocates a directory for
it. This is of the form `/boot/loader/entries/ostree-$osname-$checksum.$serial.conf`.
he $serial is normally 0, but if a
given commit is deployed more than once, it will be incremented.
his is supported because the previous deployment may have
configuration in `/etc`
hat we do not want to use or overwrite.
Now that we have a deployment directory, a 3-way merge is
performed between the (by default) currently booted deployment's
`/etc`, its default
configuration, and the new deployment (based on its `/usr/etc`).
## Atomically swapping boot configuration
At this point, a new deployment directory has been created as a
hardlink farm; the running system is untouched, and the bootloader
configuration is untouched. We want to add this deployment o the
"deployment list".
To support a more general case, OSTree supports atomic ransitioning
between arbitrary sets of deployments, with the restriction that the
currently booted deployment must always be in the new set. In the
normal case, we have exactly one deployment, which is the booted one,
and we want to add the new deployment to the list. A more complex
command might allow creating 100 deployments as part of one atomic
transaction, so that one can set up an automated system to bisect
across them.
## The bootversion
OSTree allows swapping between boot configurations by implementing the
"swapped directory pattern" in `/boot`. This means it is a symbolic
link to one of two directories `/ostree/boot.[0|1]`. To swap the
contents atomically, if the current version is `0`, we create
`/ostree/boot.1`, populate it with the new contents, then atomically
swap the symbolic link. Finally, the old contents can be garbage
collected at any point.
## The /ostree/boot directory
However, we want to optimize for the case where we the set of
kernel/initramfs pairs is the same between both the old and new
deployment lists. This happens when doing an upgrade that does not
include the kernel; think of a simple translation update. OSTree
optimizes for this case because on some systems `/boot` may be on a
separate medium such as flash storage not optimized for significant
amounts of write traffic.
To implement this, OSTree also maintains the directory
`/ostree/boot.<replaceable>bootversion</replaceable>`, which is a set
of symbolic links to the deployment directories. The
<replaceable>bootversion</replaceable> here must match the version of
`/boot`. However, in order to allow atomic transitions of
<emphasis>this</emphasis> directory, this is also a swapped directory,
so just like `/boot`, it has a version of `0` or `1` appended.
Each bootloader entry has a special `ostree=` argument which refers to
one of these symbolic links. This is parsed at runtime in the
initramfs.
docs/manual/deployment.md
+90
View File
@@ -0,0 +1,90 @@
# Deployments
## Overview
Built on top of the OSTree versioning filesystem core is a layer
that knows how to deploy, parallel install, and manage Unix-like
operating systems (accessible via `ostree admin`). The core content of these operating systems
are treated as read-only, but they transparently share storage.
A deployment is physically located at a path of the form
`/ostree/deploy/$osname/deploy/$checksum`.
OSTree is designed to boot directly into exactly one deployment
at a time; each deployment is intended to be a target for
`chroot()` or equivalent.
### "osname": Group of deployments that share /var</title>
Each deployment is grouped in exactly one "osname". From above, you
can see that an osname is physically represented in the
`/ostree/deploy/$osname` directory. For example, OSTree can allow
parallel installing Debian in `/ostree/deploy/debian` and Red Hat
Enterprise Linux in `/ostree/deploy/rhel` (subject to operating system
support, present released versions of these operating systems may not
support this).
Each osname has exactly one copy of the traditional Unix `/var`,
stored physically in `/ostree/deploy/$osname/var`. OSTree provides
support tools for `systemd` to create a Linux bind mount that ensures
the booted deployment sees the shared copy of `/var`.
OSTree does not touch the contents of `/var`. Operating system
components such as daemon services are required to create any
directories they require there at runtime
(e.g. `/var/cache/$daemonname`), and to manage upgrading data formats
inside those directories.
### Contents of a deployment
A deployment begins with a specific commit (represented as a
SHA256 hash) in the OSTree repository in `/ostree/repo`. This commit refers
to a filesystem tree that represents the underlying basis of a
deployment. For short, we will call this the "tree", to
distinguish it from the concept of a deployment.
First, the tree must include a kernel stored as
`/boot/vmlinuz-$checksum`. The checksum should be a SHA256 hash of
the kernel contents; it must be pre-computed before storing the kernel
in the repository. Optionally, the tree can contain an initramfs,
stored as `/boot/initramfs-$checksum`. If this exists, the checksum
must include both the kernel and initramfs contents. OSTree will use
this to determine which kernels are shared. The rationale for this is
to avoid computing checksums on the client by default.
The deployment should not have a traditional UNIX `/etc`; instead, it
should include `/usr/etc`. This is the "default configuration". When
OSTree creates a deployment, it performs a 3-way merge using the
<emphasis>old</emphasis> default configuration, the active system's
`/etc`, and the new default configuration. In the final filesystem
tree for a deployment then, `/etc` is a regular writable directory.
Besides the exceptions of `/var` and `/etc` then, the rest of the
contents of the tree are checked out as hard links into the
repository. It's strongly recommended that operating systems ship all
of their content in `/usr`, but this is not a hard requirement.
Finally, a deployment may have a `.origin` file, stored next to its
directory. This file tells `ostree admin upgrade` how to upgrade it.
At the moment, OSTree only supports upgrading a single refspec.
However, in the future OSTree may support a syntax for composing
layers of trees, for example.
### The system /boot
While OSTree parallel installs deployments cleanly inside the
`/ostree` directory, ultimately it has to control the system's `/boot`
directory. The way this works is via the
[Boot Loader Specification](http://www.freedesktop.org/wiki/Specifications/BootLoaderSpec),
which is a standard for bootloader-independent drop-in configuration
files.
When a tree is deployed, it will have a configuration file generated
of the form
`/boot/loader/entries/ostree-$osname-$checksum.$serial.conf`. This
configuration file will include a special `ostree=` kernel argument
that allows the initramfs to find (and `chroot()` into) the specified
deployment.
At present, not all bootloaders implement the BootLoaderSpec, so
OSTree contains code for some of these to regenerate native config
files (such as `/boot/syslinux/syslinux.conf` based on the entries.
docs/manual/introduction.md
+110
View File
@@ -0,0 +1,110 @@
# OSTree Overview
## Introduction
OSTree an upgrade system for Linux-based operating systems that
performs atomic upgrades of complete filesystem trees. It is
not a package system; rather, it is intended to complement them.
A primary model is composing packages on a server, and then
replicating them to clients.
The underlying architecture might be summarized as "git for
operating system binaries". It operates in userspace, and will
work on top of any Linux filesystem. At its core is a git-like
content-addressed object store, and layered on top of that is
bootloader configuration, management of
`/etc`, and other functions to perform an
upgrade beyond just replicating files.
You can use OSTree standalone in the pure replication model,
but another approach is to add a package manager on top,
thus creating a hybrid tree/package system.
## Comparison with "package managers"
Because OSTree is designed for deploying core operating
systems, a comparison with traditional "package managers" such
as dpkg and rpm is illustrative. Packages are traditionally
composed of partial filesystem trees with metadata and scripts
attached, and these are dynamically assembled on the client
machine, after a process of dependency resolution.
In contrast, OSTree only supports recording and deploying
*complete* (bootable) filesystem trees. It
has no built-in knowledge of how a given filesystem tree was
generated or the origin of individual files, or dependencies,
descriptions of individual components. Put another way, OSTree
only handles delivery and deployment; you will likely still want
to include inside each tree metadata about the individual
components that went into the tree. For example, a system
administrator may want to know what version of OpenSSL was
included in your tree, so you should support the equivalent of
`rpm -q` or `dpkg -L`.
The OSTree core emphasizes replicating read-only OS trees via
HTTP, and where the OS includes (if desired) an entirely
separate mechanism to install applications, stored in `/var` if they're system global, or
`/home` for per-user
application installation. An example application mechanism is
http://docker.io/
However, it is entirely possible to use OSTree underneath a
package system, where the contents of `/usr` are computed on the client.
For example, when installing a package, rather than changing the
currently running filesystem, the package manager could assemble
a new filesystem tree that layers the new packages on top of a
base tree, record it in the local OSTree repository, and then
set it up for the next boot. To support this model, OSTree
provides an (introspectable) C shared library.
## Comparison with block/image replication
OSTree shares some similarity with "dumb" replication and
stateless deployments, such as the model common in "cloud"
deployments where nodes are booted from an (effectively)
readonly disk, and user data is kept on a different volumes.
The advantage of "dumb" replication, shared by both OSTree and
the cloud model, is that it's *reliable*
and *predictable*.
But unlike many default image-based deployments, OSTree supports
exactly two persistent writable directories that are preserved across
upgrades: `/etc` and `/var`.
Because OSTree operates at the Unix filesystem layer, it works
on top of any filesystem or block storage layout; it's possible
to replicate a given filesystem tree from an OSTree repository
into plain ext4, BTRFS, XFS, or in general any Unix-compatible
filesystem that supports hard links. Note: OSTree will
transparently take advantage of some BTRFS features if deployed
on it.
## Atomic transitions between parallel-installable read-only filesystem trees
Another deeply fundamental difference between both package
managers and image-based replication is that OSTree is
designed to parallel-install *multiple versions* of multiple
*independent* operating systems. OSTree
relies on a new toplevel `ostree` directory; it can in fact
parallel install inside an existing OS or distribution
occupying the physical `/` root.
On each client machine, there is an OSTree repository stored
in `/ostree/repo`, and a set of "deployments" stored in `/ostree/deploy/$OSNAME/$CHECKSUM`.
Each deployment is primarily composed of a set of hardlinks
into the repository. This means each version is deduplicated;
an upgrade process only costs disk space proportional to the
new files, plus some constant overhead.
The model OSTree emphasizes is that the OS read-only content
is kept in the classic Unix `/usr`; it comes with code to
create a Linux read-only bind mount to prevent inadvertent
corruption. There is exactly one `/var` writable directory shared
between each deployment for a given OS. The OSTree core code
does not touch content in this directory; it is up to the code
in each operating system for how to manage and upgrade state.
Finally, each deployment has its own writable copy of the
configuration store `/etc`. On upgrade, OSTree will
perform a basic 3-way diff, and apply any local changes to the
new copy, while leaving the old untouched.
docs/manual/repo.md
+81
View File
@@ -0,0 +1,81 @@
# Anatomy of an OSTree repository
## Core object types and data model
OSTree is deeply inspired by git; the core layer is a userspace
content-addressed versioning filesystem. It is worth taking some time
to familiarize yourself with
[Git Internals](http://git-scm.com/book/en/Git-Internals), as this
section will assume some knowledge of how git works.
Its object types are similar to git; it has commit objects and content
objects. Git has "tree" objects, whereas OSTree splits them into
"dirtree" and "dirmeta" objects. But unlike git, OSTree's checksums
are SHA256. And most crucially, its content objects include uid, gid,
and extended attributes (but still no timestamps).
### Commit objects
A commit object contains metadata such as a timestamp, a log
message, and most importantly, a reference to a
dirtree/dirmeta pair of checksums which describe the root
directory of the filesystem.
Also like git, each commit in OSTree can have a parent. It is
designed to store a history of your binary builds, just like git
stores a history of source control. However, OSTree also makes
it easy to delete data, under the assumption that you can
regenerate it from source code.
### Dirtree objects
A dirtree contains a sorted array of (filename, checksum)
pairs for content objects, and a second sorted array of
(filename, dirtree checksum, dirmeta checksum), which are
subdirectories.
### Dirmeta objects
In git, tree objects contain the metadata such as permissions
for their children. But OSTree splits this into a separate
object to avoid duplicating extended attribute listings.
### Content objects
Unlike the first three object types which are metadata, designed to be
`mmap()`ed, the content object has a separate internal header and
payload sections. The header contains uid, gid, mode, and symbolic
link target (for symlinks), as well as extended attributes. After the
header, for regular files, the content follows.
# Repository types and locations
Also unlike git, an OSTree repository can be in one of two separate
modes: `bare` and `archive-z2`. A bare repository is one where
content files are just stored as regular files; it's designed to be
the source of a "hardlink farm", where each operating system checkout
is merely links into it. If you want to store files owned by
e.g. root in this mode, you must run OSTree as root. In contrast, the
`archive-z2` mode is designed for serving via plain HTTP. Like tar
files, it can be read/written by non-root users.
On an OSTree-deployed system, the "system repository" is
`/ostree/repo`. It can be read by any uid, but only written by root.
Unless the `--repo` argument is given to the <command>ostree</command>
command, it will operate on the system repository.
## Refs
Like git, OSTree uses "refs" to which are text files that point to
particular commits (i.e. filesystem trees). For example, the
gnome-ostree operating system creates trees named like
`exampleos/buildmaster/x86_64-runtime` and
`exampleos/buildmaster/x86_64-devel-debug`. These two refs point to
two different generated filesystem trees. In this example, the
"runtime" tree contains just enough to run a basic system, and
"devel-debug" contains all of the developer tools and debuginfo.
The `ostree` supports a simple syntax using the carat `^` to refer to
the parent of a given commit. For example,
`exampleos/buildmaster/x86_64-runtime^` refers to the previous build,
and `exampleos/buildmaster/x86_64-runtime^^` refers to the one before
that.