packaging
/
ostree
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ostree/src/libostree
History
Philip Withnall e7305bbc8a lib/repo-pull: Prefer object pull over from-scratch delta if ref exists 
If a ref already exists, we are likely only a few commits behind the
current head of the ref, so it is probably better for bandwidth
consumption to pull the individual objects rather than the from-scratch
delta.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1709
Approved by: cgwalters
 		2018-08-20 13:04:58 +00:00
..
README-gpg pull: Verify commits with gpg signatures from detached metadata 2013-09-29 14:49:47 -04:00
README.md core: Associate branches with remotes, move trigger runs into checkout 2012-04-03 23:46:34 -04:00
bupsplit.c lib: Move the bupsplit selftest into our test framework 2017-02-02 16:51:36 +00:00
bupsplit.h lib: Move the bupsplit selftest into our test framework 2017-02-02 16:51:36 +00:00
libostree-devel.sym Post-release version bump 2018-07-20 19:43:18 +00:00
libostree-released.sym Release 2018.7 2018-07-20 19:43:18 +00:00
ostree-1.pc.in libostree: Expose $OSTREE_FEATURES in the pkg-config file 2017-05-08 18:48:07 +00:00
ostree-async-progress.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-async-progress.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-autocleanups.h Revert "lib: Fix building against old glib versions" 2018-06-07 17:28:49 +00:00
ostree-bloom-private.h Fix building P2P code against old glib versions 2018-05-30 14:19:25 +00:00
ostree-bloom.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-bootconfig-parser.c tree-wide: Remove Emacs modelines 2017-09-21 21:38:34 +00:00
ostree-bootconfig-parser.h tree-wide: Remove Emacs modelines 2017-09-21 21:38:34 +00:00
ostree-bootloader-grub2.c tree-wide: Update to new libglnx fd APIs 2017-10-11 19:26:10 +00:00
ostree-bootloader-grub2.h tree-wide: Remove Emacs modelines 2017-09-21 21:38:34 +00:00
ostree-bootloader-syslinux.c syslinux: Add support for DEVICETREE from bootloader spec 2018-01-16 22:54:53 +00:00
ostree-bootloader-syslinux.h tree-wide: Remove Emacs modelines 2017-09-21 21:38:34 +00:00
ostree-bootloader-uboot.c u-boot: add support for devicetree 2018-05-23 12:25:38 +00:00
ostree-bootloader-uboot.h tree-wide: Remove Emacs modelines 2017-09-21 21:38:34 +00:00
ostree-bootloader.c tree-wide: Remove Emacs modelines 2017-09-21 21:38:34 +00:00
ostree-bootloader.h tree-wide: Remove Emacs modelines 2017-09-21 21:38:34 +00:00
ostree-chain-input-stream.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-chain-input-stream.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-checksum-input-stream.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-checksum-input-stream.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-cmdprivate.c Add concept of "staged" deployment 2018-04-12 14:55:12 +00:00
ostree-cmdprivate.h Add concept of "staged" deployment 2018-04-12 14:55:12 +00:00
ostree-core-private.h Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
ostree-core.c Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
ostree-core.h Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
ostree-deployment-private.h Add concept of "staged" deployment 2018-04-12 14:55:12 +00:00
ostree-deployment.c Add concept of "staged" deployment 2018-04-12 14:55:12 +00:00
ostree-deployment.h Add concept of "staged" deployment 2018-04-12 14:55:12 +00:00
ostree-diff.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-diff.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-dummy-enumtypes.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-dummy-enumtypes.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-enumtypes.c.template build: Add -Werror=undef by default, fix fallout 2017-11-06 15:30:14 +00:00
ostree-enumtypes.h.template Make enum generation private by default, export stub symbol 2016-05-12 17:15:50 +00:00
ostree-fetcher-curl.c lib/fetcher: Factor out HTTP status code handling from soup and curl 2018-05-30 16:23:57 +00:00
ostree-fetcher-soup.c lib/fetcher: Factor out HTTP status code handling from soup and curl 2018-05-30 16:23:57 +00:00
ostree-fetcher-uri.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-fetcher-util.c lib/fetcher: Factor out HTTP status code handling from soup and curl 2018-05-30 16:23:57 +00:00
ostree-fetcher-util.h lib/fetcher: Factor out HTTP status code handling from soup and curl 2018-05-30 16:23:57 +00:00
ostree-fetcher.h lib/fetcher: Allow clients to append to User-Agent 2018-03-16 19:21:31 +00:00
ostree-gpg-verifier.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-gpg-verifier.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-gpg-verify-result-private.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-gpg-verify-result.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-gpg-verify-result.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-impl-system-generator.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-kernel-args.c tree-wide: Use autoptr for OstreeKernelArgs 2017-10-25 01:48:56 +00:00
ostree-kernel-args.h tree-wide: Use autoptr for OstreeKernelArgs 2017-10-25 01:48:56 +00:00
ostree-libarchive-input-stream.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-libarchive-input-stream.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-libarchive-private.h lib/archive: Tell g-ir-scanner to ignore the private libarchive bits 2018-06-18 16:40:39 +00:00
ostree-linuxfsutil.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-linuxfsutil.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-lzma-common.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-lzma-common.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-lzma-compressor.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-lzma-compressor.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-lzma-decompressor.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-lzma-decompressor.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-metalink.c lib/repo-pull: Support retrying requests on transient network errors 2018-05-30 16:23:57 +00:00
ostree-metalink.h lib/repo-pull: Support retrying requests on transient network errors 2018-05-30 16:23:57 +00:00
ostree-mutable-tree.c ostree repo commit: Speed up composing trees with `--tree=ref` 2018-07-09 13:10:51 +00:00
ostree-mutable-tree.h ostree repo commit: Speed up composing trees with `--tree=ref` 2018-07-09 13:10:51 +00:00
ostree-ref.c Revert "lib: Fix building against old glib versions" 2018-06-07 17:28:49 +00:00
ostree-ref.h Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
ostree-remote-private.h Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
ostree-remote.c Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
ostree-remote.h Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
ostree-repo-checkout.c repo: Create uncompressed-object-cache dir dynamically 2018-02-08 21:27:11 +00:00
ostree-repo-commit.c lib/commit: Only auto-update summary if refs were written 2018-08-01 19:59:07 +00:00
ostree-repo-deprecated.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-file-enumerator.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-file-enumerator.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-file.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-file.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-finder-avahi-parser.c Fix building against old glib versions 2018-06-07 17:28:49 +00:00
ostree-repo-finder-avahi-private.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-finder-avahi.c avahi: Fail immediately if we can't talk to D-Bus or Avahi 2018-06-23 14:54:39 +00:00
ostree-repo-finder-avahi.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-finder-config.c OstreeRepoFinderConfig: Fix guint/gsize confusion 2018-06-27 15:24:18 +00:00
ostree-repo-finder-config.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-finder-mount.c Revert "lib: Fix building against old glib versions" 2018-06-07 17:28:49 +00:00
ostree-repo-finder-mount.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-finder-override.c Revert "lib: Fix building against old glib versions" 2018-06-07 17:28:49 +00:00
ostree-repo-finder-override.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-finder.c Fix building against old glib versions 2018-06-07 17:28:49 +00:00
ostree-repo-finder.h Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
ostree-repo-libarchive.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-private.h repo: Add OSTREE_REPO_TEST_ERROR=invalid-cache env var 2018-08-14 13:38:11 +00:00
ostree-repo-prune.c lib/prune: Don't modify dirent->d_name in place 2018-06-15 19:01:46 +00:00
ostree-repo-pull-private.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-pull.c lib/repo-pull: Prefer object pull over from-scratch delta if ref exists 2018-08-20 13:04:58 +00:00
ostree-repo-refs.c Fix leak in ostree_repo_list_collection_refs 2018-08-13 16:29:59 +00:00
ostree-repo-static-delta-compilation-analysis.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-static-delta-compilation.c ostree_repo_static_delta_generate: Fix leak 2018-08-13 17:50:33 +00:00
ostree-repo-static-delta-core.c lib/deltas: Some misc declare-and-initialize porting 2018-03-23 18:16:35 +00:00
ostree-repo-static-delta-private.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-static-delta-processing.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-repo-traverse.c lib/repo: Add ostree_repo_traverse_commit_union_with_parents 2018-04-14 15:36:21 +00:00
ostree-repo.c repo: remove outdated note from write_config() docs 2018-08-19 02:01:12 +00:00
ostree-repo.h Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
ostree-rollsum.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-rollsum.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-sepolicy-private.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-sepolicy.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-sepolicy.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-soup-form.c lib/repo: Add (transfer) annotations to various GHashTable arguments 2017-11-14 23:13:14 +00:00
ostree-soup-uri.c lib/repo: Add (transfer) annotations to various GHashTable arguments 2017-11-14 23:13:14 +00:00
ostree-soup-uri.h libcurl backend 2017-02-09 16:37:45 +00:00
ostree-sysroot-cleanup.c lib/sysroot: Add wrapper API to prune system repository 2018-05-24 12:56:11 +00:00
ostree-sysroot-deploy.c deploy: Fix overriding kernel args for staged deployments 2018-07-09 18:04:57 +00:00
ostree-sysroot-private.h lib/sysroot: Maintain one variable for "root is ostree booted" 2018-05-02 13:27:01 +00:00
ostree-sysroot-upgrader.c lib/sysroot: Add OSTREE_EX_STAGE_DEPLOYMENTS environment variable 2018-05-11 12:39:17 +00:00
ostree-sysroot-upgrader.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-sysroot.c sysroot: Reject attempts to pin the staged deployment 2018-06-29 01:52:30 +00:00
ostree-sysroot.h lib/sysroot: Add wrapper API to prune system repository 2018-05-24 12:56:11 +00:00
ostree-tls-cert-interaction.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-tls-cert-interaction.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-types.h Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00
ostree-varint.c Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-varint.h Add SPDX-License-Identifier to source files 2018-01-30 20:03:42 +00:00
ostree-version.h.in tree-wide: Remove Emacs modelines 2017-09-21 21:38:34 +00:00
ostree.h Make P2P API public (no longer experimental) 2018-06-04 19:20:10 +00:00

README.md

Repository design

At the heart of OSTree is the repository. It's very similar to git, with the idea of content-addressed storage. However, OSTree is designed to store operating system binaries, not source code. There are several consequences to this. The key difference as compared to git is that the OSTree definition of "content" includes key Unix metadata such as owner uid/gid, as well as all extended attributes.

Essentially OSTree is designed so that if two files have the same OSTree checksum, it's safe to replace them with a hard link. This fundamental design means that an OSTree repository imposes negligible overhead. In contrast, a git repository stores copies of zlib-compressed data.

Key differences versus git

  • As mentioned above, extended attributes and owner uid/gid are versioned
  • Optimized for Unix hardlinks between repository and checkout
  • SHA256 instead of SHA1
  • Support for empty directories

Binary files

While this is still in planning, I plan to heavily optimize OSTree for versioning ELF operating systems. In industry jargon, this would be "content-aware storage".

Trimming history

OSTree will also be optimized to trim intermediate history; in theory one can regenerate binaries from corresponding (git) source code, so we don't need to keep all possible builds over time.

MILESTONE 1

  • Basic pack files (like git)

MILESTONE 2

  • Store checksums as ay
  • Drop version/metadata from tree/dirmeta objects
  • Add index size to superindex, pack size to index
    • So pull can calculate how much we need to download
  • Split pack files into metadata/data
  • pull: Extract all we can from each packfile one at a time, then delete it
  • Restructure repository so that links can be generated as a cache; i.e. objects/raw, pack files are now the canonical
  • For files, checksum combination of metadata variant + raw data
    • i.e. there is only OSTREE_OBJECT_TYPE_FILE (again)

MILESTONE 3

  • Drop archive/raw distinction - archive repositories always generate packfiles per commit
  • Include git packv4 ideas:
    • metadata packfiles have string dictionary (tree filenames and checksums)
    • data packfiles match up similar objects
  • Rolling checksums for partitioning large files? Kernel debuginfo
  • Improved pack clustering
    • file fingerprinting?
  • ELF-x86 aware deltas

Related work in storage

git: http://git-scm.com/ Venti: http://plan9.bell-labs.com/magic/man2html/6/venti Elephant FS: http://www.hpl.hp.com/personal/Alistair_Veitch/papers/elephant-hotos/index.html

Compression

xdelta: http://xdelta.org/ Bsdiff: http://www.daemonology.net/bsdiff/ xz: http://tukaani.org/xz/