Go to file
Dan Nicholson 4e78ddd2da lib/repo: Add repo locking mechanism
Currently ostree has no method of guarding against concurrent pruning.
When there are multiple repo writers, it's possible to have a pull or
commit race against a prune and end up with missing objects.

This adds a file based repo locking mechanism. The intention is to take
a shared lock when writing objects and an exclusive lock when deleting
them. In order to make use of the locking throughout the library in a
fine grained fashion, the lock acts recursively with a stack of lock
states. If the lock becomes exclusive, it will stay in that state until
the stack is unwound past the initial exclusive push. The file locking
is similar to GLnxLockFile in that it uses open file descriptor locks
but falls back to flock when needed.

The lock also attempts to be thread safe by storing the lock state in
thread local storage with GPrivate. This means that each thread will
have an independent lock for each repository it opens. There are some
drawbacks to that, but it seemed impossible to manage the lock state
coherently in the face of multithreaded access.

The API is a push/pop interface in accordance with the recursive nature
of the locking. The push interface uses an enum that's translated to
LOCK_SH or LOCK_EX as needed. Both interfaces use an internal timeout
field to decide whether to manage the lock in a blocking or non-blocking
fashion. The intention is to allow ostree applications as well as
administrators to control this timeout. For now, the default is a 30
second timeout.

Note that the timeout is handled synchronously in thread since the lock
is maintained in thread local storage. I.e., the thread that acquires
the lock needs to be the same thread that runs the operation. There may
be a way to offer an asynchronous version, but it's not clear exactly
how that would work since it would likely involve a separate thread that
invokes a callback when the locking operation completes.

https://bugzilla.gnome.org/show_bug.cgi?id=759442

Closes: #1343
Approved by: cgwalters
2017-12-05 02:32:47 +00:00
apidoc lib/repo: Add repo locking mechanism 2017-12-05 02:32:47 +00:00
bash lib/core: add ostree_checksum_file_at API 2017-10-12 12:53:01 +00:00
bsdiff@1edf9f6568 bsdiff: change submodule location 2015-03-26 23:33:07 +01:00
build-aux Add infrastructure for "make syntax-check" 2015-01-30 15:27:36 +01:00
buildutil tests: allow specifying tmpdir 2017-09-21 21:50:40 +00:00
ci ci: Really show test-suite.log on travis 2017-12-01 19:00:18 +00:00
coccinelle tree-wide: Add+run spatch to use glnx_throw() 2017-05-26 19:27:11 +00:00
docs docs/related-projects: Add Google Usenix link 2017-10-21 14:12:02 +00:00
libglnx@b36606b366 build: Add -Werror=undef by default, fix fallout 2017-11-06 15:30:14 +00:00
man lib/core: add ostree_checksum_file_at API 2017-10-12 12:53:01 +00:00
manual-tests tests/delta-crosscheck: Disable fsync 2017-12-01 15:09:14 +00:00
rust rust: Support `make dist` -> cargo vendor 2017-02-07 16:50:37 +00:00
src lib/repo: Add repo locking mechanism 2017-12-05 02:32:47 +00:00
tests Release 2017.14 2017-12-04 16:41:06 +00:00
.dir-locals.el .dir-locals.el: Standard Emacs indentation config 2017-01-12 16:09:34 +00:00
.editorconfig Add a .vimrc and .editorconfig 2017-09-21 22:03:11 +00:00
.gitmodules bsdiff: change submodule location 2015-03-26 23:33:07 +01:00
.papr.yml papr: Bump primary to f27 2017-11-27 17:46:07 +00:00
.travis.yml ci: Move travis scripts from tests/ → ci/ 2017-05-09 18:25:13 +00:00
.vimrc Add a .vimrc and .editorconfig 2017-09-21 22:03:11 +00:00
CONTRIBUTING.md Rewrite manual in mkdocs 2016-01-28 09:31:37 -05:00
COPYING COPYING: Update to latest FSF with current address 2014-01-16 10:22:30 -05:00
GNUmakefile Add infrastructure for "make syntax-check" 2015-01-30 15:27:36 +01:00
Makefile-bash.am build: Add distcheck configure flags to fix systemd and bash-completion 2017-08-22 17:37:48 +00:00
Makefile-boot.am build: Ensure ostree-tmpfiles.conf is distributed 2017-08-22 17:37:48 +00:00
Makefile-decls.am build-sys: Minor makefile tweaks 2017-01-04 16:32:11 +00:00
Makefile-libostree-defines.am lib/repo-finder: Add OstreeRepoFinderOverride 2017-10-19 19:11:58 +00:00
Makefile-libostree.am build: Add a TODO comment about improving glib-mkenums usage in future 2017-11-07 15:14:56 +00:00
Makefile-man.am build: Don't distribute generated man pages 2017-07-18 21:51:43 +00:00
Makefile-ostree.am create-usb: Add a create-usb command to complement OstreeRepoFinderMount 2017-09-27 14:44:00 +00:00
Makefile-otutil.am Add OtVariantBuilder 2017-10-27 21:49:26 +00:00
Makefile-switchroot.am build: Add distcheck configure flags to fix systemd and bash-completion 2017-08-22 17:37:48 +00:00
Makefile-tests.am tests: Run python tests with stdout unbuffered 2017-12-01 19:00:18 +00:00
Makefile.am build: Also fix -Werror=undef for old libsoup 2017-11-06 21:32:30 +00:00
README-historical.md README: Just link to wiki, move most of it to README-historical.md 2014-01-20 18:00:09 -05:00
README.md README.md: Link to BuildStream 2017-12-04 18:26:51 +00:00
TODO Fix repeated words. 2015-01-30 15:27:36 +01:00
autogen.sh Make sure *.am.inc are up to date before `make dist` 2017-10-16 13:53:06 +00:00
cfg.mk syntax-check: Add a rule to enforce glnx_autofd over glnx_fd_close 2017-10-17 16:43:02 +00:00
configure.ac build-sys: Post-release version bump 2017-12-04 18:11:45 +00:00
git.mk Use git.mk 2016-04-07 12:49:40 +00:00
maint.mk build/maint.mk: Comment out setting of LC_ALL 2017-08-23 17:41:06 +00:00
mkdocs.yml docs: Add a section on repository management 2016-03-29 14:10:24 +00:00
ostree.doap doap category infrastructure 2014-07-31 11:26:32 +02:00

README.md

libostree

New! See the docs online at Read The Docs (OSTree)


This project is now known as "libostree", though it is still appropriate to use the previous name: "OSTree" (or "ostree"). The focus is on projects which use libostree's shared library, rather than users directly invoking the command line tools (except for build systems). However, in most of the rest of the documentation, we will use the term "OSTree", since it's slightly shorter, and changing all documentation at once is impractical. We expect to transition to the new name over time.

As implied above, libostree is both a shared library and suite of command line tools that combines a "git-like" model for committing and downloading bootable filesystem trees, along with a layer for deploying them and managing the bootloader configuration.

The core OSTree model is like git in that it checksums individual files and has a content-addressed-object store. It's unlike git in that it "checks out" the files via hardlinks, and they should thus be immutable. Therefore, another way to think of OSTree is that it's just a more polished version of Linux VServer hardlinks.

Features:

  • Transactional upgrades and rollback for the system
  • Replicating content incrementally over HTTP via GPG signatures and "pinned TLS" support
  • Support for parallel installing more than just 2 bootable roots
  • Binary history on the server side (and client)
  • Introspectable shared library API for build and deployment systems
  • Flexible support for multiple branches and repositories, supporting projects like flatpak which use libostree for applications, rather than hosts.

Projects using OSTree

meta-updater is a layer available for OpenEmbedded systems.

QtOTA is Qt's over-the-air update framework which uses libostree.

rpm-ostree is a next-generation hybrid package/image system for Fedora and CentOS, used by the Atomic Host project. By default it uses libostree to atomically replicate a base OS (all dependency resolution is done on the server), but it supports "package layering", where additional RPMs can be layered on top of the base. This brings a "best of both worlds"" model for image and package systems.

flatpak uses libostree for desktop application containers. Unlike most of the other systems here, flatpak does not use the "libostree host system" aspects (e.g. bootloader management), just the "git-like hardlink dedup". For example, flatpak supports a per-user OSTree repository.

Endless OS uses libostree for their host system as well as flatpak. See their eos-updater and deb-ostree-builder projects.

GNOME Continuous is where OSTree was born - as a high performance continuous delivery/testing system for GNOME.

The BuildStream build and integration tool uses libostree as a caching system to store and share built artifacts.

Building

Releases are available as GPG signed git tags, and most recent versions support extended validation using git-evtag.

However, in order to build from a git clone, you must update the submodules. If you're packaging OSTree and want a tarball, I recommend using a "recursive git archive" script. There are several available online; this code in OSTree is an example.

Once you have a git clone or recursive archive, building is the same as almost every autotools project:

git submodule update --init
env NOCONFIGURE=1 ./autogen.sh
./configure --prefix=...
make
make install DESTDIR=/path/to/dest

More documentation

New! See the docs online at Read The Docs (OSTree)

Contributing

See Contributing.