Go to file
William Manley c7b12a8730 ostree repo commit: Speed up composing trees with `--tree=ref`
Running `ostree commit --tree=ref=a --tree=dir=b` involves reading the
whole of a into an `OstreeMutableTree` before composing `b` on top.  This
is inefficient if a is a complete rootfs and b is just touching one file.
We process O(size of a + size of b) directories rather than
O(number of touched dirs).

This commit makes `ostree commit` more efficient at composing multiple
directories together.  With `ostree_mutable_tree_fill_empty_from_dirtree`
we create a lazy `OstreeMutableTree` which only reads the underlying
information from disk when needed.  We don't need to read all the
subdirectories just to get the checksum of a tree already checked into the
repo.

This provides great speedups when composing a rootfs out of multiple other
rootfs as we do in our build system.  We compose multiple containers
together with:

    ostree commit --tree=ref=base-rootfs --tree=ref=container1 --tree=ref=container2

and it is much faster now.

As a test I ran

    time ostree --repo=... commit --orphan --tree=ref=big-rootfs --tree=dir=modified_etc

Where modified_etc contained a modified sudoers file under /etc.  I used
`strace` to count syscalls and I seperatly took timing measurements.  To
test with a cold cache I ran

    sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

Results:

|                      | Before | After |
| -------------------- | ------ | ----- |
| Time (cold cache)    |   8.1s | 0.12s |
| Time (warm cache)    |   3.7s | 0.08s |
| `openat` calls       |  53589 |   246 |
| `fgetxattr` calls    |  78916 |     0 |

I'm not sure if this will have some negative interaction with the
`_ostree_repo_commit_modifier_apply` which is short-circuited here.  I
think it was disabled for `--tree=ref=` anyway, but I'm not certain.  All
the tests pass anyway.

I originally implemented this in terms of the `OstreeRepoFile` APIs, but
it was *way* less efficient, opening and reading many files unnecessarily.

Closes: #1643
Approved by: cgwalters
2018-07-09 13:10:51 +00:00
apidoc ostree repo commit: Speed up composing trees with `--tree=ref` 2018-07-09 13:10:51 +00:00
bash ostree/pull: Add network-retries command line option 2018-07-05 17:59:09 +00:00
bsdiff@1edf9f6568 bsdiff: change submodule location 2015-03-26 23:33:07 +01:00
build-aux Add infrastructure for "make syntax-check" 2015-01-30 15:27:36 +01:00
buildutil Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ci ci/flatpak: Fix to use built ostree version 2018-06-29 20:54:42 +00:00
coccinelle tree-wide: Add+run spatch to use glnx_throw() 2017-05-26 19:27:11 +00:00
docs docs: var/local -> var/usrlocal 2018-05-15 16:01:05 +00:00
libglnx@e1a78cf2f5 Update libglnx to get g_autoptr backports 2018-05-29 19:24:20 +00:00
man ostree/pull: Add network-retries command line option 2018-07-05 17:59:09 +00:00
manual-tests Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
rust rust/bupsplit: minor idiomatic fixes 2018-03-17 19:59:06 +00:00
src ostree repo commit: Speed up composing trees with `--tree=ref` 2018-07-09 13:10:51 +00:00
tests deploy: Retain staged by default 2018-07-06 15:23:52 +00:00
.dir-locals.el .dir-locals.el: Standard Emacs indentation config 2017-01-12 16:09:34 +00:00
.editorconfig Add a .vimrc and .editorconfig 2017-09-21 22:03:11 +00:00
.gitmodules .gitmodules: Update URL for libglnx 2018-05-30 13:48:48 +00:00
.papr-ex.yaml ci: exclude 'lib' from libsoup configure option 2018-06-29 13:16:05 +00:00
.papr.yml ci: exclude 'lib' from libsoup configure option 2018-06-29 13:16:05 +00:00
.travis.yml ci: Move travis scripts from tests/ → ci/ 2017-05-09 18:25:13 +00:00
.vimrc Add a .vimrc and .editorconfig 2017-09-21 22:03:11 +00:00
CONTRIBUTING.md Rewrite manual in mkdocs 2016-01-28 09:31:37 -05:00
COPYING COPYING: Update to latest FSF with current address 2014-01-16 10:22:30 -05:00
GNUmakefile Add infrastructure for "make syntax-check" 2015-01-30 15:27:36 +01:00
Makefile-bash.am Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
Makefile-boot.am Add concept of "staged" deployment 2018-04-12 14:55:12 +00:00
Makefile-decls.am Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
Makefile-libostree-defines.am Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
Makefile-libostree.am build: add ostree-soup-* to build process when configured with avahi 2018-07-03 19:42:49 +00:00
Makefile-man.am Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
Makefile-ostree.am Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
Makefile-otutil.am Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
Makefile-switchroot.am build: Use ostree_prepare_root_CPPFLAGS for ostree-prepare-root 2018-07-04 19:45:50 +00:00
Makefile-tests.am deploy: Delete .updated file from /etc and /var on new deployments 2018-06-18 13:21:52 +00:00
Makefile.am Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
README-historical.md README: Just link to wiki, move most of it to README-historical.md 2014-01-20 18:00:09 -05:00
README.md docs: Add "Hello World" example 2018-05-08 14:49:59 +00:00
TODO Fix repeated words. 2015-01-30 15:27:36 +01:00
autogen.sh Make sure *.am.inc are up to date before `make dist` 2017-10-16 13:53:06 +00:00
cfg.mk tests: Add a test case for path traversal in a dirtree 2018-01-12 19:38:34 +00:00
configure.ac Post-release version bump 2018-06-21 11:23:40 -04:00
git.mk Use git.mk 2016-04-07 12:49:40 +00:00
maint.mk build/maint.mk: Comment out setting of LC_ALL 2017-08-23 17:41:06 +00:00
mkdocs.yml docs: Add a section on repository management 2016-03-29 14:10:24 +00:00
ostree.doap doap category infrastructure 2014-07-31 11:26:32 +02:00

README.md

libostree

New! See the docs online at Read The Docs (OSTree)


This project is now known as "libostree", though it is still appropriate to use the previous name: "OSTree" (or "ostree"). The focus is on projects which use libostree's shared library, rather than users directly invoking the command line tools (except for build systems). However, in most of the rest of the documentation, we will use the term "OSTree", since it's slightly shorter, and changing all documentation at once is impractical. We expect to transition to the new name over time.

As implied above, libostree is both a shared library and suite of command line tools that combines a "git-like" model for committing and downloading bootable filesystem trees, along with a layer for deploying them and managing the bootloader configuration.

The core OSTree model is like git in that it checksums individual files and has a content-addressed-object store. It's unlike git in that it "checks out" the files via hardlinks, and they thus need to be immutable to prevent corruption. Therefore, another way to think of OSTree is that it's just a more polished version of Linux VServer hardlinks.

Features:

  • Transactional upgrades and rollback for the system
  • Replicating content incrementally over HTTP via GPG signatures and "pinned TLS" support
  • Support for parallel installing more than just 2 bootable roots
  • Binary history on the server side (and client)
  • Introspectable shared library API for build and deployment systems
  • Flexible support for multiple branches and repositories, supporting projects like flatpak which use libostree for applications, rather than hosts.

Projects using OSTree

meta-updater is a layer available for OpenEmbedded systems.

QtOTA is Qt's over-the-air update framework which uses libostree.

rpm-ostree is a next-generation hybrid package/image system for Fedora and CentOS, used by the Atomic Host project. By default it uses libostree to atomically replicate a base OS (all dependency resolution is done on the server), but it supports "package layering", where additional RPMs can be layered on top of the base. This brings a "best of both worlds"" model for image and package systems.

flatpak uses libostree for desktop application containers. Unlike most of the other systems here, flatpak does not use the "libostree host system" aspects (e.g. bootloader management), just the "git-like hardlink dedup". For example, flatpak supports a per-user OSTree repository.

Endless OS uses libostree for their host system as well as flatpak. See their eos-updater and deb-ostree-builder projects.

GNOME Continuous is where OSTree was born - as a high performance continuous delivery/testing system for GNOME.

The BuildStream build and integration tool uses libostree as a caching system to store and share built artifacts.

Building

Releases are available as GPG signed git tags, and most recent versions support extended validation using git-evtag.

However, in order to build from a git clone, you must update the submodules. If you're packaging OSTree and want a tarball, I recommend using a "recursive git archive" script. There are several available online; this code in OSTree is an example.

Once you have a git clone or recursive archive, building is the same as almost every autotools project:

git submodule update --init
env NOCONFIGURE=1 ./autogen.sh
./configure --prefix=...
make
make install DESTDIR=/path/to/dest

More documentation

New! See the docs online at Read The Docs (OSTree)

Contributing

See Contributing.

Licensing

The licensing for the code of libostree can be canonically found in the individual files; and the overall status in the COPYING file in the source. Currently, that's LGPLv2+. This also covers the man pages and API docs.

The license for the manual documentation in the doc/ directory is: SPDX-License-Identifier: (CC-BY-SA-3.0 OR GFDL-1.3-or-later) This is intended to allow use by Wikipedia and other projects.

In general, files should have a SPDX-License-Identifier and that is canonical.