packaging
/
ostree
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ostree/src/libostree
History
Philip Withnall fbf8df8829 lib/refs: Add methods for setting/listing collection–refs 
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
 		2017-06-26 15:56:07 +00:00
..
README-gpg pull: Verify commits with gpg signatures from detached metadata 2013-09-29 14:49:47 -04:00
README.md core: Associate branches with remotes, move trigger runs into checkout 2012-04-03 23:46:34 -04:00
bupsplit.c lib: Move the bupsplit selftest into our test framework 2017-02-02 16:51:36 +00:00
bupsplit.h lib: Move the bupsplit selftest into our test framework 2017-02-02 16:51:36 +00:00
libostree-devel.sym lib/core: Add ostree_validate_remote_name() for remote names 2017-06-20 21:52:22 +00:00
libostree-experimental.sym lib/refs: Add methods for setting/listing collection–refs 2017-06-26 15:56:07 +00:00
libostree-released.sym Release 2017.7 2017-06-19 14:54:59 +00:00
ostree-1.pc.in libostree: Expose $OSTREE_FEATURES in the pkg-config file 2017-05-08 18:48:07 +00:00
ostree-async-progress.c libostree: Fix potential use of uninitialised memory in progress API 2017-05-05 14:58:03 +00:00
ostree-async-progress.h libostree: Add multiple getter/setter support to OstreeAsyncProgress 2017-04-29 11:50:15 +00:00
ostree-autocleanups.h lib/ref: Add OstreeCollectionRef type for globally unique refs 2017-06-26 15:56:07 +00:00
ostree-bootconfig-parser.c lib/boot: Convert bootconfig parser to new code style 2017-04-19 13:26:20 +00:00
ostree-bootconfig-parser.h lib: Introduce versioned symbols 2016-03-01 21:45:26 -05:00
ostree-bootloader-grub2.c tree-wide: Convert to using autoptr(GString) vs g_string_free(...,TRUE) 2017-05-05 15:10:51 +00:00
ostree-bootloader-grub2.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-bootloader-syslinux.c libglnx porting: Port "load utf8 file" contents API callers 2016-05-30 11:33:28 +00:00
ostree-bootloader-syslinux.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-bootloader-uboot.c u-boot: Merge ostree's and systems uEnv.txt 2016-08-27 13:11:22 +00:00
ostree-bootloader-uboot.h libostree: Make OstreeBootloader private 2013-09-15 20:26:13 -04:00
ostree-bootloader.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-bootloader.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-chain-input-stream.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-chain-input-stream.h lib: Introduce versioned symbols 2016-03-01 21:45:26 -05:00
ostree-checksum-input-stream.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-checksum-input-stream.h lib: Introduce versioned symbols 2016-03-01 21:45:26 -05:00
ostree-cmdprivate.c Switch to using a systemd generator for /var 2017-05-16 16:13:05 +00:00
ostree-cmdprivate.h Switch to using a systemd generator for /var 2017-05-16 16:13:05 +00:00
ostree-core-private.h lib/ref: Add OstreeCollectionRef type for globally unique refs 2017-06-26 15:56:07 +00:00
ostree-core.c lib/ref: Add OstreeCollectionRef type for globally unique refs 2017-06-26 15:56:07 +00:00
ostree-core.h lib/ref: Add OstreeCollectionRef type for globally unique refs 2017-06-26 15:56:07 +00:00
ostree-deployment-private.h docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-deployment.c Rename "osname" → "stateroot" 2017-04-24 16:09:51 +00:00
ostree-deployment.h admin: Add an `unlock` command, and libostree API 2016-03-23 11:09:09 -04:00
ostree-diff.c diff: Port some to new code style 2017-05-09 18:33:12 +00:00
ostree-diff.h diff: Add ostree_diff_dirs_with_options(), expose via cmdline 2017-03-21 13:38:04 +00:00
ostree-dummy-enumtypes.c build: Don't run glib-mkenums if there's nothing to do 2016-05-12 17:15:50 +00:00
ostree-dummy-enumtypes.h build: Don't run glib-mkenums if there's nothing to do 2016-05-12 17:15:50 +00:00
ostree-enumtypes.c.template lib: Ensure we use _GNU_SOURCE in enum templates 2016-12-07 23:00:58 +00:00
ostree-enumtypes.h.template Make enum generation private by default, export stub symbol 2016-05-12 17:15:50 +00:00
ostree-fetcher-curl.c fetcher: Send Accept-Encoding: gzip when downloading summary 2017-05-24 13:52:17 +00:00
ostree-fetcher-soup.c fetcher: Send Accept-Encoding: gzip when downloading summary 2017-05-24 13:52:17 +00:00
ostree-fetcher-uri.c libcurl backend 2017-02-09 16:37:45 +00:00
ostree-fetcher-util.c fetcher: Log failures into journal 2017-02-24 22:30:24 +00:00
ostree-fetcher-util.h fetcher: Log failures into journal 2017-02-24 22:30:24 +00:00
ostree-fetcher.h lib/fetcher: Add cleanup function for OstreeFetcher 2017-06-07 16:55:36 +00:00
ostree-gpg-verifier.c repo: Delete the last use of GFile tmp_dir 2017-05-01 16:44:59 +00:00
ostree-gpg-verifier.h Add "gpgkeypath" option to remotes 2016-11-17 09:44:07 +00:00
ostree-gpg-verify-result-private.h libotutil: Establish a place for GPG utilities 2015-05-01 10:20:34 -04:00
ostree-gpg-verify-result.c tree-wide: Add+run spatch to use glnx_throw() 2017-05-26 19:27:11 +00:00
ostree-gpg-verify-result.h repo: Factor out the check of gpg result to a separate function 2016-05-27 11:20:00 +00:00
ostree-impl-system-generator.c switchroot/generator: Add var.mount to local-fs.target.requires 2017-05-17 13:53:54 +00:00
ostree-kernel-args.c Filter bootloader supplied kernel cmdline options 2016-11-01 18:00:37 +00:00
ostree-kernel-args.h Filter bootloader supplied kernel cmdline options 2016-11-01 18:00:37 +00:00
ostree-libarchive-input-stream.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-libarchive-input-stream.h Define and use cleanup helpers for libarchive 2016-12-05 15:20:56 +00:00
ostree-libarchive-private.h libostree: added empty ot_cleanup_{read,write}_archive macros. 2017-02-10 14:06:37 +00:00
ostree-linuxfsutil.c lib/fsutil: Port to new code style 2017-05-26 19:05:53 +00:00
ostree-linuxfsutil.h lib/fsutil: Delete unused GFile ioctl method 2017-05-26 19:05:53 +00:00
ostree-lzma-common.c static-delta: Handle LZMA_BUF_ERROR returned by zlib 2015-10-10 10:27:17 -04:00
ostree-lzma-common.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-lzma-compressor.c docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-lzma-compressor.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-lzma-decompressor.c docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-lzma-decompressor.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-metalink.c fetcher: Move high level functions into "fetcher-util" 2017-01-04 16:32:11 +00:00
ostree-metalink.h fetcher: Define an abstraction over SoupURI 2016-12-07 23:00:58 +00:00
ostree-mutable-tree.c docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-mutable-tree.h docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-ref.c lib/ref: Add OstreeCollectionRef type for globally unique refs 2017-06-26 15:56:07 +00:00
ostree-ref.h lib/ref: Add OstreeCollectionRef type for globally unique refs 2017-06-26 15:56:07 +00:00
ostree-remote-private.h lib/remote: Add arguments to internal OstreeRemote constructor 2017-05-19 15:01:59 +00:00
ostree-remote.c lib/remote: Add arguments to internal OstreeRemote constructor 2017-05-19 15:01:59 +00:00
ostree-remote.h lib/remote: Fix ‘Since’ line for OstreeRemote 2017-06-21 13:00:55 +00:00
ostree-repo-checkout.c checkout: Fix SELinux policy labeling when recursing 2017-06-16 14:54:29 +00:00
ostree-repo-commit.c lib/refs: Add methods for setting/listing collection–refs 2017-06-26 15:56:07 +00:00
ostree-repo-deprecated.h libostree: mark ostree_repo_checkout_tree_at as deprecated 2016-07-30 11:24:52 +00:00
ostree-repo-file-enumerator.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-repo-file-enumerator.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-repo-file.c lib/repofile: Follow symlinks for `g_file_read()` 2017-06-07 20:54:32 +00:00
ostree-repo-file.h lib: Remove ostree_repo_file_make_empty_tree 2016-04-17 13:52:07 +00:00
ostree-repo-libarchive.c tree-wide: Add+run spatch to use glnx_throw() 2017-05-26 19:27:11 +00:00
ostree-repo-private.h lib/refs: Add methods for setting/listing collection–refs 2017-06-26 15:56:07 +00:00
ostree-repo-prune.c lib/refs: Add methods for setting/listing collection–refs 2017-06-26 15:56:07 +00:00
ostree-repo-pull.c pull: When mirroring, only replace summary if we're doing a full mirror 2017-06-16 15:08:39 +00:00
ostree-repo-refs.c lib/refs: Add methods for setting/listing collection–refs 2017-06-26 15:56:07 +00:00
ostree-repo-static-delta-compilation-analysis.c [ASAN] lib: Squash various leaks in library and commandline 2016-11-21 16:34:06 +00:00
ostree-repo-static-delta-compilation.c lib/deltas: Port to more to new code style 2017-06-26 14:42:56 +00:00
ostree-repo-static-delta-core.c tree-wide: Add+run spatch to use glnx_throw() 2017-05-26 19:27:11 +00:00
ostree-repo-static-delta-private.h lib: Always checksum content in deltas 2016-12-06 15:59:35 +00:00
ostree-repo-static-delta-processing.c lib/repo: More cleanup of load_file() internals 2017-06-23 18:29:51 +00:00
ostree-repo-traverse.c traverse: Use g_hash_table_add 2016-11-30 18:51:26 +00:00
ostree-repo.c lib/refs: Add methods for setting/listing collection–refs 2017-06-26 15:56:07 +00:00
ostree-repo.h lib/refs: Add methods for setting/listing collection–refs 2017-06-26 15:56:07 +00:00
ostree-rollsum.c rollsum: Fix assertion for CRC matches 2015-08-25 09:26:22 -04:00
ostree-rollsum.h lib/deltas: Port to more to new code style 2017-06-26 14:42:56 +00:00
ostree-sepolicy-private.h checkout: Add SELinux labeling for checkout, use in deploy 2017-04-25 16:52:33 +00:00
ostree-sepolicy.c libglnx: bump and use new helper methods 2017-05-12 21:02:16 +00:00
ostree-sepolicy.h sepolicy: Add ostree_sepolicy_new_at() 2017-03-22 16:24:06 +00:00
ostree-soup-form.c libcurl backend 2017-02-09 16:37:45 +00:00
ostree-soup-uri.c libcurl backend 2017-02-09 16:37:45 +00:00
ostree-soup-uri.h libcurl backend 2017-02-09 16:37:45 +00:00
ostree-sysroot-cleanup.c lib/sysroot: Add some g_prefix_error() for ostree_sysroot_cleanup() 2017-06-15 00:08:29 +00:00
ostree-sysroot-deploy.c tree-wide: Add+run spatch to use glnx_throw() 2017-05-26 19:27:11 +00:00
ostree-sysroot-private.h Revert "Add a notion of "physical" sysroot, use for remote writing" 2017-06-02 14:11:58 +00:00
ostree-sysroot-upgrader.c Allow commits to mark refs as EOL, replaced by others 2017-05-22 20:18:07 +00:00
ostree-sysroot-upgrader.h upgrade: Add support for --pull-only and --deploy-only 2017-02-27 20:57:23 +00:00
ostree-sysroot.c lib/sysroot: Add API to get pending/rollback for given stateroot 2017-06-14 09:56:01 +00:00
ostree-sysroot.h lib/sysroot: Add API to get pending/rollback for given stateroot 2017-06-14 09:56:01 +00:00
ostree-tls-cert-interaction.c fetcher: Rework API to use strings for tls keys/db 2017-01-19 10:47:15 +00:00
ostree-tls-cert-interaction.h fetcher: Rework API to use strings for tls keys/db 2017-01-19 10:47:15 +00:00
ostree-types.h libostree: Make OstreeRemote a public and internal API 2017-05-08 18:48:07 +00:00
ostree-varint.c syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-varint.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-version.h.in ostreee-version.h.in: Added Since: version annotations 2017-06-15 12:59:59 +00:00
ostree.h lib/ref: Add OstreeCollectionRef type for globally unique refs 2017-06-26 15:56:07 +00:00

README.md

Repository design

At the heart of OSTree is the repository. It's very similar to git, with the idea of content-addressed storage. However, OSTree is designed to store operating system binaries, not source code. There are several consequences to this. The key difference as compared to git is that the OSTree definition of "content" includes key Unix metadata such as owner uid/gid, as well as all extended attributes.

Essentially OSTree is designed so that if two files have the same OSTree checksum, it's safe to replace them with a hard link. This fundamental design means that an OSTree repository imposes negligible overhead. In contrast, a git repository stores copies of zlib-compressed data.

Key differences versus git

  • As mentioned above, extended attributes and owner uid/gid are versioned
  • Optimized for Unix hardlinks between repository and checkout
  • SHA256 instead of SHA1
  • Support for empty directories

Binary files

While this is still in planning, I plan to heavily optimize OSTree for versioning ELF operating systems. In industry jargon, this would be "content-aware storage".

Trimming history

OSTree will also be optimized to trim intermediate history; in theory one can regenerate binaries from corresponding (git) source code, so we don't need to keep all possible builds over time.

MILESTONE 1

  • Basic pack files (like git)

MILESTONE 2

  • Store checksums as ay
  • Drop version/metadata from tree/dirmeta objects
  • Add index size to superindex, pack size to index
    • So pull can calculate how much we need to download
  • Split pack files into metadata/data
  • pull: Extract all we can from each packfile one at a time, then delete it
  • Restructure repository so that links can be generated as a cache; i.e. objects/raw, pack files are now the canonical
  • For files, checksum combination of metadata variant + raw data
    • i.e. there is only OSTREE_OBJECT_TYPE_FILE (again)

MILESTONE 3

  • Drop archive/raw distinction - archive repositories always generate packfiles per commit
  • Include git packv4 ideas:
    • metadata packfiles have string dictionary (tree filenames and checksums)
    • data packfiles match up similar objects
  • Rolling checksums for partitioning large files? Kernel debuginfo
  • Improved pack clustering
    • file fingerprinting?
  • ELF-x86 aware deltas

Related work in storage

git: http://git-scm.com/ Venti: http://plan9.bell-labs.com/magic/man2html/6/venti Elephant FS: http://www.hpl.hp.com/personal/Alistair_Veitch/papers/elephant-hotos/index.html

Compression

xdelta: http://xdelta.org/ Bsdiff: http://www.daemonology.net/bsdiff/ xz: http://tukaani.org/xz/