packaging
/
ostree
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ostree/src/libostree
History
Colin Walters 886913abdc sysroot: Add ostree_sysroot_write_origin_file() API 
We want to allow admins to change the origin file without doing a new
deployment, so this will be part of a future "admin set-origin"
command.
 		2015-01-19 13:45:11 -05:00
..
README-deltas.md Initial basic static delta code drop 2014-02-04 10:31:44 -05:00
README-gpg pull: Verify commits with gpg signatures from detached metadata 2013-09-29 14:49:47 -04:00
README.md core: Associate branches with remotes, move trigger runs into checkout 2012-04-03 23:46:34 -04:00
bupsplit.c core: Import bup's "rollsum" code, add a test case 2014-02-04 10:12:56 -05:00
bupsplit.h core: Import bup's "rollsum" code, add a test case 2014-02-04 10:12:56 -05:00
ostree-1.pc.in Install a shared library 2013-07-26 19:25:07 -04:00
ostree-async-progress.c libostree: Add _finish() API to async progress 2014-04-29 10:59:57 -04:00
ostree-async-progress.h libostree: Add _finish() API to async progress 2014-04-29 10:59:57 -04:00
ostree-bootconfig-parser.c deploy: fdatasync() bootloader configuration files 2014-04-11 18:43:27 -04:00
ostree-bootconfig-parser.h libostree: Add ostree_bootconfig_parser_clone() 2014-03-19 09:49:55 -04:00
ostree-bootloader-grub2.c trivial: Fix typo in previous commit 2014-11-25 13:47:54 -05:00
ostree-bootloader-grub2.h Add (non-atomic) support for GRUB2 + UEFI 2014-10-23 21:42:30 -04:00
ostree-bootloader-syslinux.c syslinux: Adapt to bootloader UI changes 2014-11-10 21:31:32 -05:00
ostree-bootloader-syslinux.h libostree: Make OstreeBootloader private 2013-09-15 20:26:13 -04:00
ostree-bootloader-uboot.c Add (non-atomic) support for GRUB2 + UEFI 2014-10-23 21:42:30 -04:00
ostree-bootloader-uboot.h libostree: Make OstreeBootloader private 2013-09-15 20:26:13 -04:00
ostree-bootloader.c Add (non-atomic) support for GRUB2 + UEFI 2014-10-23 21:42:30 -04:00
ostree-bootloader.h Add (non-atomic) support for GRUB2 + UEFI 2014-10-23 21:42:30 -04:00
ostree-chain-input-stream.c libostree: Fix many gtk-doc warnings 2013-08-17 08:41:31 -04:00
ostree-chain-input-stream.h core: Fix all introspection warnings 2013-07-27 10:13:30 -04:00
ostree-checksum-input-stream.c libostree: Fix many gtk-doc warnings 2013-08-17 08:41:31 -04:00
ostree-checksum-input-stream.h Switch to #pragma once for headers 2013-07-09 18:53:22 -04:00
ostree-cmdprivate.c Add (non-atomic) support for GRUB2 + UEFI 2014-10-23 21:42:30 -04:00
ostree-cmdprivate.h make ostree_cmd__private__ ret value const 2014-10-21 16:18:30 -04:00
ostree-core-private.h Merge branch 'giuseppe/staticdeltas' of https://github.com/giuseppe/ostree 2014-12-19 16:31:31 -05:00
ostree-core.c core: Add an API to parse a content file using dirfd relative lookup 2015-01-14 22:03:02 -05:00
ostree-core.h core: Add an API to parse a content file using dirfd relative lookup 2015-01-14 22:03:02 -05:00
ostree-deployment.c libostree: Fix ostree_deployment_clone() to also clone bootconfig 2014-03-19 09:49:55 -04:00
ostree-deployment.h libostree: Fix an introspection scanner warning 2013-10-09 12:53:12 -04:00
ostree-diff.c Allow creating and pulling static deltas starting from "empty" 2015-01-14 14:43:32 +01:00
ostree-diff.h SELinux: Ensure we label /var, and fix /etc merge wrt xattrs 2014-02-02 11:32:52 -05:00
ostree-fetcher.c Change OstreeFetcher to be dirfd-relative 2015-01-14 22:12:08 -05:00
ostree-fetcher.h Change OstreeFetcher to be dirfd-relative 2015-01-14 22:12:08 -05:00
ostree-gpg-verifier.c libostree: Add a better error if we fail to read keyring directory 2014-03-14 14:42:46 -04:00
ostree-gpg-verifier.h trivial: Don't include config.h in headers 2013-10-15 15:26:37 -04:00
ostree-kernel-args.c ostree admin instutil set-kargs: make more flexible 2014-09-26 10:40:37 -04:00
ostree-kernel-args.h ostree admin instutil set-kargs: make more flexible 2014-09-26 10:40:37 -04:00
ostree-libarchive-input-stream.c libostree: Make OstreeLibarchiveInputStream explicitly private 2014-06-09 14:04:53 -04:00
ostree-libarchive-input-stream.h libostree: Make OstreeLibarchiveInputStream explicitly private 2014-06-09 14:04:53 -04:00
ostree-linuxfsutil.c libostree: Silently ignore EPERM when setting EXT2_IMMUTABLE_FL 2014-06-09 14:44:17 -04:00
ostree-linuxfsutil.h deploy: Set the immutable bit on the deployment root 2014-06-03 13:41:48 -04:00
ostree-lzma-common.c deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-lzma-common.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-lzma-compressor.c deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-lzma-compressor.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-lzma-decompressor.c deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-lzma-decompressor.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-metalink.c Change OstreeFetcher to be dirfd-relative 2015-01-14 22:12:08 -05:00
ostree-metalink.h Change OstreeFetcher to be dirfd-relative 2015-01-14 22:12:08 -05:00
ostree-mutable-tree.c doc: Add OstreeMutableTree, other minor doc updates 2013-09-12 23:53:13 -04:00
ostree-mutable-tree.h Switch to #pragma once for headers 2013-07-09 18:53:22 -04:00
ostree-repo-checkout.c checkout: Add --fsync=false 2015-01-07 11:41:46 -05:00
ostree-repo-commit.c Add internal ot_openat_read_stream() helper 2015-01-07 08:41:45 -05:00
ostree-repo-file-enumerator.c Install a shared library 2013-07-26 19:25:07 -04:00
ostree-repo-file-enumerator.h Switch to #pragma once for headers 2013-07-09 18:53:22 -04:00
ostree-repo-file.c repofile: Avoid segfault if querying child in non-directory 2014-07-21 07:58:07 -04:00
ostree-repo-file.h repo-file: Base OstreeRepoFile on trees instead of commits 2013-09-09 22:31:59 -04:00
ostree-repo-libarchive.c Enforce 'rdev' (device file major/minor) is 0 2014-12-19 10:20:05 -05:00
ostree-repo-private.h Static deltas support 2014-12-18 12:48:47 +01:00
ostree-repo-prune.c repo-pull: Allow pulling only one directory 2014-08-20 15:09:32 -04:00
ostree-repo-pull.c pull: Delete processed delta parts 2015-01-14 22:18:00 -05:00
ostree-repo-refs.c refs: Use G_IO_ERROR_NOT_FOUND when a ref lookup fails 2015-01-06 18:47:04 -05:00
ostree-repo-static-delta-compilation.c Allow creating and pulling static deltas starting from "empty" 2015-01-14 14:43:32 +01:00
ostree-repo-static-delta-core.c Fix ostree_repo_list_static_delta_names 2015-01-14 14:36:35 +01:00
ostree-repo-static-delta-private.h Static deltas support 2014-12-18 12:48:47 +01:00
ostree-repo-static-delta-processing.c deltas: Use *at() for writes 2015-01-14 22:29:45 -05:00
ostree-repo-traverse.c [staticanalysis]: Fix two uses of uninitialized variables 2014-06-21 18:08:49 -04:00
ostree-repo.c Allow creating and pulling static deltas starting from "empty" 2015-01-14 14:43:32 +01:00
ostree-repo.h Merge branch 'giuseppe/staticdeltas' of https://github.com/giuseppe/ostree 2014-12-19 16:31:31 -05:00
ostree-sepolicy.c Port to libgsystem errno API, hard depend on 2014.3 2015-01-04 21:17:11 -05:00
ostree-sepolicy.h libostree: Split off SELinux OstreeSePolicy class 2014-02-19 08:43:45 -05:00
ostree-sysroot-cleanup.c deploy: Set the immutable bit on the deployment root 2014-06-03 13:41:48 -04:00
ostree-sysroot-deploy.c sysroot: Add ostree_sysroot_write_origin_file() API 2015-01-19 13:45:11 -05:00
ostree-sysroot-private.h Add (non-atomic) support for GRUB2 + UEFI 2014-10-23 21:42:30 -04:00
ostree-sysroot-upgrader.c Add "unconfigured-state" concept to origin files 2014-10-03 14:03:55 -04:00
ostree-sysroot-upgrader.h Add "unconfigured-state" concept to origin files 2014-10-03 14:03:55 -04:00
ostree-sysroot.c Add (non-atomic) support for GRUB2 + UEFI 2014-10-23 21:42:30 -04:00
ostree-sysroot.h sysroot: Add ostree_sysroot_write_origin_file() API 2015-01-19 13:45:11 -05:00
ostree-tls-cert-interaction.c libostree: Make OstreeTlsCertInteraction explicitly private 2014-06-09 14:04:53 -04:00
ostree-tls-cert-interaction.h libostree: Make OstreeTlsCertInteraction explicitly private 2014-06-09 14:04:53 -04:00
ostree-types.h Add an OstreeSysrootUpgrader API 2014-03-24 18:08:22 -04:00
ostree-varint.c Initial basic static delta code drop 2014-02-04 10:31:44 -05:00
ostree-varint.h Initial basic static delta code drop 2014-02-04 10:31:44 -05:00
ostree.h Add an OstreeSysrootUpgrader API 2014-03-24 18:08:22 -04:00

README.md

Repository design

At the heart of OSTree is the repository. It's very similar to git, with the idea of content-addressed storage. However, OSTree is designed to store operating system binaries, not source code. There are several consequences to this. The key difference as compared to git is that the OSTree definition of "content" includes key Unix metadata such as owner uid/gid, as well as all extended attributes.

Essentially OSTree is designed so that if two files have the same OSTree checksum, it's safe to replace them with a hard link. This fundamental design means that an OSTree repository imposes negligible overhead. In contrast, a git repository stores copies of zlib-compressed data.

Key differences versus git

  • As mentioned above, extended attributes and owner uid/gid are versioned
  • Optimized for Unix hardlinks between repository and checkout
  • SHA256 instead of SHA1
  • Support for empty directories

Binary files

While this is still in planning, I plan to heavily optimize OSTree for versioning ELF operating systems. In industry jargon, this would be "content-aware storage".

Trimming history

OSTree will also be optimized to trim intermediate history; in theory one can regenerate binaries from corresponding (git) source code, so we don't need to keep all possible builds over time.

MILESTONE 1

  • Basic pack files (like git)

MILESTONE 2

  • Store checksums as ay
  • Drop version/metadata from tree/dirmeta objects
  • Add index size to superindex, pack size to index
    • So pull can calculate how much we need to download
  • Split pack files into metadata/data
  • pull: Extract all we can from each packfile one at a time, then delete it
  • Restructure repository so that links can be generated as a cache; i.e. objects/raw, pack files are now the canonical
  • For files, checksum combination of metadata variant + raw data
    • i.e. there is only OSTREE_OBJECT_TYPE_FILE (again)

MILESTONE 3

  • Drop archive/raw distinction - archive repositories always generate packfiles per commit
  • Include git packv4 ideas:
    • metadata packfiles have string dictionary (tree filenames and checksums)
    • data packfiles match up similar objects
  • Rolling checksums for partitioning large files? Kernel debuginfo
  • Improved pack clustering
    • file fingerprinting?
  • ELF-x86 aware deltas

Related work in storage

git: http://git-scm.com/ Venti: http://plan9.bell-labs.com/magic/man2html/6/venti Elephant FS: http://www.hpl.hp.com/personal/Alistair_Veitch/papers/elephant-hotos/index.html

Compression

xdelta: http://xdelta.org/ Bsdiff: http://www.daemonology.net/bsdiff/ xz: http://tukaani.org/xz/