Imported Upstream version 2016.6

docs/manual/atomic-upgrades.md

@@ -45,7 +45,7 @@ operate "live" on the currently booted filesystem.  The way they could
 work with OSTree is instead to take the list of installed packages in
 the currently booted tree, and compute a new filesystem from that.  A
 later chapter describes in more details how this could work:
-[adapting-existing.md](Adapting Existing Systems).
+[Adapting Existing Systems](adapting-existing.md).
 
 For the purposes of this section, let's assume that we have a
 newly generated filesystem tree stored in the repo (which shares
@@ -56,7 +56,7 @@ checking it back out of the repo into a deployment.
 
 Given a commit to deploy, OSTree first allocates a directory for
 it.  This is of the form `/boot/loader/entries/ostree-$osname-$checksum.$serial.conf`.
-The `$serial` is normally 0, but if a
+The `$serial` is normally `0`, but if a
 given commit is deployed more than once, it will be incremented.
 This is supported because the previous deployment may have
 configuration in `/etc` that we do not want to use or overwrite.

docs/manual/formats.md

@@ -125,8 +125,7 @@ the client executes.
 
 This "updates as code" model allows for multiple content generation
 strategies.  The design of this was inspired by that of Chromium:
-[http://dev.chromium.org/chromium-os/chromiumos-design-docs/filesystem-autoupdate](ChromiumOS
-autoupdate).
+[ChromiumOS Autoupdate](http://dev.chromium.org/chromium-os/chromiumos-design-docs/filesystem-autoupdate).
 
 ### The delta superblock
 

docs/manual/related-projects.md

@@ -78,6 +78,10 @@ available.
 All of the above also applies if one replaces "BTRFS" with "LVM
 snapshots" except for the reflinks.
 
+OSTree supports using "bare-user" repositories, which do not require
+root to use. Using a filesystem-level layer without root is more
+difficult and would likely require a setuid helper or privileged service.
+
 Finally, see the next portion around ChromiumOS for why a hybrid but
 integrated package/image system improves on this.
 
@@ -130,6 +134,11 @@ believes that at the moment, the "CL updater" is not truly atomic in
 the sense that because it applies updates live, there is a window
 where the OS root may be inconsistent.
 
+## Mender.io
+
+[Mender.io](https://mender.io/) is another implementation of the dual
+partition approach.
+
 ## OLPC update
 
 OSTree is basically a generalization of olpc-update, except using
@@ -147,37 +156,54 @@ See
 [this comment](http://blog.verbum.org/2013/08/26/ostree-v2013-6-released/#comment-1169)
 for a comparison.
 
-## NixOS
+## NixOS / Nix
 
-See [NixOS](http://nixos.org/). It was a very influential project for
-OSTree.  NixOS and OSTree both support the idea of independent "roots"
-that are bootable.
+See [NixOS](http://nixos.org/). It was a very influential project for OSTree.
+NixOS and OSTree both support the idea of independent "roots" that are bootable.
 
-In NixOS, the entire system is based on checksums of package inputs
-(build dependencies) - see [Nix store](http://nixos.org/nix/manual/#chap-package-management/). A both
-positive and negative of the Nix model is that a change in the build
-dependencies (e.g. being built with a newer gcc), requires a cascading
-rebuild of everything.
+In NixOS, files in a package are accessed by a path depending on the checksums
+of package inputs (build dependencies) - see
+[Nix store](http://nixos.org/nix/manual/#chap-package-management/).
+However, OSTree uses a commit/deploy model - it isn't tied to any particular
+directory layout, and you can put whatever data you want inside an OSTree, for
+example the standard FHS layout. A both positive and negative of the Nix model
+is that a change in the build dependencies (e.g. being built with a newer gcc),
+requires a cascading rebuild of everything. It's good because it makes it easy
+to do massive system-wide changes such as gcc upgrades, and allows installing
+multiple versions of packages at once. However, a security update to e.g. glibc
+forces a rebuild of everything from scratch, and so Nix is not practical at
+scale. OSTree supports using a build system that just rebuilds individual
+components (packages) as they change, without forcing a rebuild of their
+dependencies.
 
-In OSTree, the checksums are of object *content* (including extended
-attributes). This means that any data that's identical is
-transparently, automatically shared on disk. It's possible to ask the
-Nix store to deduplicate, (via hard links and immutable bit), but this
-is significantly less efficient than the OSTree approach. The Nix use
-of the ext immutable bit is racy, since it has to be briefly removed
-to make a hard link.
+Nix automatically detects runtime package dependencies by scanning content for
+hashes. OSTree only supports only system-level images, and doesn't do dependency
+management. Nix can store arbitrary files, using nix-store --add, but, more
+commonly, paths are added as the result of running a derivation file generated
+using the Nix language. OSTree is build-system agnostic; filesystem trees are
+committed using a simple C API, and this is the only way to commit files.
 
-At the lowest level, OSTree is just "git for binaries" - it isn't tied
-strongly to any particular build system. You can put whatever data you
-want inside an OSTree repository, built however you like. So for
-example, while one could make a build system that did the "purely
-functional" approach of Nix, it also works to have a build system that
-just rebuilds individual components (packages) as they change, without
-forcing a rebuild of their dependencies.
+OSTree automatically shares the storage of identical data using hard links into
+a content-addressed store. Nix can deduplicate using hard links as well, using
+the auto-optimise-store option, but this is not on by default, and Nix does not
+guarantee that all of its files are in the content-addressed store. OSTree
+provides a git-like command line interface for browsing the content-addressed
+store, while Nix does not have this functionality.
 
-The author of OSTree believes that while Nix has some good ideas,
-forcing a rebuild of everything for a security update to e.g. glibc is
-not practical at scale.
+Nix used to use the immutable bit to prevent modifications to /nix/store, but
+now it uses a read-only bind mount. The bind mount can be privately remounted,
+allowing per-process privileged write access. OSTree uses the immutable
+bit on the root of the deployment, and mounts /usr as read-only.
+
+NixOS supports switching OS images on-the-fly, by maintaining both booted-system
+and current-system roots. It is not clear how well this approach works. OSTree
+currently requries a reboot to switch images.
+
+Finally, NixOS supports installing user-specific packages from trusted
+repositories without requiring root, using a trusted daemon.
+[Flatpak](https://lwn.net/Articles/687909/), based on OSTree, similarly has a
+policykit-based system helper that allows you to authenticate via polkit to
+install into the system repository.
 
 ## Solaris IPS
 
@@ -205,3 +231,40 @@ See
 [bmap](https://source.tizen.org/documentation/reference/bmaptool/introduction).
 A tool for optimized copying of disk images. Intended for offline use,
 so not directly comparable.
+
+## Git
+
+Although OSTree has been called "Git for Binaries", and the two share the idea
+of a hashed content store, the implementation details are quite different.
+OSTree supports extended attributes and uses SHA256 instead of Git's SHA1. It
+"checks out" files via hardlinks, rather than copying, and thus requires the
+checkout to be immutable. At the moment, OSTree commits may have at most one
+parent, as opposed to Git which allows an arbitrary number. Git uses a
+smart-delta protocol for updates, while OSTree uses 1 HTTP request per changed
+file, or can generate static deltas.
+
+## Conda
+
+[Conda](http://conda.pydata.org/docs/) is an "OS-agnostic, system-level binary
+package manager and ecosystem"; although most well-known for its accompanying
+Python distribution anaconda, its scope has been expanding quickly. The package
+format is very similar to well-known ones such as RPM. However, unlike typical
+RPMs, the packages are built to be relocatable. Also, the package manager runs
+natively on Windows. Conda's main advantage is its ability to install
+collections of packages into "environments" by unpacking them all to the same
+directory. Conda reduces duplication across environments using hardlinks,
+similar to OSTree's sharing between deployments (although Conda uses package /
+file path instead of file hash). Overall, it is quite similar to rpm-ostree in
+functionality and scope.
+
+## rpm-ostree
+
+This builds on top of ostree to support building RPMs into OSTree images, and
+even composing RPMs on-the-fly using an overlay filesystem. It is being
+developed by Fedora, Red Hat, and CentOS as part of Project Atomic.
+
+## GNOME Continuous
+
+This is a service that incrementally rebuilds and tests GNOME on every commit.
+The need to make and distribute snapshots for this system was the original
+inspiration for ostree.

docs/manual/repo.md

@@ -47,6 +47,22 @@ payload sections.  The header contains uid, gid, mode, and symbolic
 link target (for symlinks), as well as extended attributes.  After the
 header, for regular files, the content follows.
 
+The OSTree data format intentionally does not contain timestamps. The reasoning
+is that data files may be downloaded at different times, and by different build
+systems, and so will have different timestamps but identical physical content.
+These files may be large, so most users would like them to be shared, both in
+the repository and between the repository and deployments.
+
+This could cause problems with programs that check if files are out-of-date by
+comparing timestamps. For Git, the logical choice is to not mess with
+timestamps, because unnecessary rebuilding is better than a broken tree.
+However, OSTree has to hardlink files to check them out, and commits are assumed
+to be internally consistent with no build steps needed. For this reason, OSTree
+acts as though all timestamps are set to time_t 1, so that comparisons will be
+considered up-to-date. 1 is a better choice than 0 because some programs use 0
+as a special value; for example, GNU Tar warns of an "implausibly old time
+stamp" with 0.
+
 # Repository types and locations
 
 Also unlike git, an OSTree repository can be in one of three separate
@@ -98,7 +114,7 @@ that.
 A later addition to OSTree is the concept of a "summary" file, created
 via the `ostree summary -u` command.  This was introduced for a few
 reasons.  A primary use case is to be a target a
-(Metalink)[https://en.wikipedia.org/wiki/Metalink], which requires a
+[Metalink](https://en.wikipedia.org/wiki/Metalink), which requires a
 single file with a known checksum as a target.
 
 The summary file primarily contains two mappings:

docs/manual/repository-management.md

@@ -4,11 +4,12 @@ Once you have a build system going, if you actually want client
 systems to retrieve the content, you will quickly feel a need for
 "repository management".
 
-OSTree itself does not currently come with tools to do this.  One
-reason is that how content is delivered and managed has concerns very
-specific to the organization.  For example, some operating system
-content vendors may want integration with a specific errata
-notification system.
+The command line tool `ostree` does cover some core functionality, but
+doesn't include very high level workflows.  One reason is that how
+content is delivered and managed has concerns very specific to the
+organization.  For example, some operating system content vendors may
+want integration with a specific errata notification system when
+generating commits.
 
 In this section, we will describe some high level ideas and methods
 for managing content in OSTree repositories, mostly independent of any
@@ -21,6 +22,27 @@ repositories today is the [Pulp Project](http://www.pulpproject.org/),
 which has a
 [Pulp OSTree plugin](https://pulp-ostree.readthedocs.org/en/latest/).
 
+## Mirroring repositories
+
+It's very common to want to perform a full or partial mirror, in
+particular across organizational boundaries (e.g. an upstream OS
+provider, and a user that wants offline and faster access to the
+content).  OSTree supports both full and partial mirroring of the base
+`archive-z2` content, although not yet of static deltas.
+
+To create a mirror, first create an `archive-z2` repository (you don't
+need to run this as root), then add the upstream as a remote, then use
+`pull --mirror`.
+
+```
+ostree --repo=repo init --mode=archive-z2
+ostree --repo=repo remote add exampleos https://exampleos.com/ostree/repo
+ostree --repo=repo pull --mirror exampleos:exampleos/x86_64/standard
+```
+
+You can use the `--depth=-1` option to retrieve all history, or a
+positive integer like `3` to retrieve just the last 3 commits.
+
 ## Separate development vs release repositories
 
 By default, OSTree accumulates server side history.  This is actually