packaging
/
ostree
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ostree/src/libostree
History
Colin Walters e09fc83ab3 lib/core: Add #defines for ref/collection binding 
These were previously private, but since we expect people to use them, let's add
`#define`s like we did for some of the other commit metadata.

Closes: #1028
Approved by: jlebon
 		2017-07-26 18:52:04 +00:00
..
README-gpg pull: Verify commits with gpg signatures from detached metadata 2013-09-29 14:49:47 -04:00
README.md core: Associate branches with remotes, move trigger runs into checkout 2012-04-03 23:46:34 -04:00
bupsplit.c lib: Move the bupsplit selftest into our test framework 2017-02-02 16:51:36 +00:00
bupsplit.h lib: Move the bupsplit selftest into our test framework 2017-02-02 16:51:36 +00:00
libostree-devel.sym build-sys: Post-release version bump 2017-07-06 14:45:02 +00:00
libostree-experimental.sym lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
libostree-released.sym Release 2017.8 2017-07-06 14:45:02 +00:00
ostree-1.pc.in libostree: Expose $OSTREE_FEATURES in the pkg-config file 2017-05-08 18:48:07 +00:00
ostree-async-progress.c libostree: Fix potential use of uninitialised memory in progress API 2017-05-05 14:58:03 +00:00
ostree-async-progress.h libostree: Add multiple getter/setter support to OstreeAsyncProgress 2017-04-29 11:50:15 +00:00
ostree-autocleanups.h lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-bloom-private.h lib/bloom: Add an internal bloom filter implementation 2017-06-26 15:56:07 +00:00
ostree-bloom.c lib/bloom: Add an internal bloom filter implementation 2017-06-26 15:56:07 +00:00
ostree-bootconfig-parser.c codebase: start using GLNX_HASH_TABLE_FOREACH macros 2017-06-28 16:37:15 +00:00
ostree-bootconfig-parser.h lib: Introduce versioned symbols 2016-03-01 21:45:26 -05:00
ostree-bootloader-grub2.c Update libglnx, port some uses to newer APIs 2017-07-24 18:43:57 +00:00
ostree-bootloader-grub2.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-bootloader-syslinux.c libglnx porting: Port "load utf8 file" contents API callers 2016-05-30 11:33:28 +00:00
ostree-bootloader-syslinux.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-bootloader-uboot.c u-boot: Merge ostree's and systems uEnv.txt 2016-08-27 13:11:22 +00:00
ostree-bootloader-uboot.h libostree: Make OstreeBootloader private 2013-09-15 20:26:13 -04:00
ostree-bootloader.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-bootloader.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-chain-input-stream.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-chain-input-stream.h lib: Introduce versioned symbols 2016-03-01 21:45:26 -05:00
ostree-checksum-input-stream.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-checksum-input-stream.h lib: Introduce versioned symbols 2016-03-01 21:45:26 -05:00
ostree-cmdprivate.c Switch to using a systemd generator for /var 2017-05-16 16:13:05 +00:00
ostree-cmdprivate.h Switch to using a systemd generator for /var 2017-05-16 16:13:05 +00:00
ostree-core-private.h lib: Add a helper to convert struct stat → GFileInfo 2017-06-29 18:17:28 +00:00
ostree-core.c Update libglnx, port some uses to newer APIs 2017-07-24 18:43:57 +00:00
ostree-core.h lib/core: Add #defines for ref/collection binding 2017-07-26 18:52:04 +00:00
ostree-deployment-private.h docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-deployment.c Rename "osname" → "stateroot" 2017-04-24 16:09:51 +00:00
ostree-deployment.h admin: Add an `unlock` command, and libostree API 2016-03-23 11:09:09 -04:00
ostree-diff.c diff: Port some to new code style 2017-05-09 18:33:12 +00:00
ostree-diff.h diff: Add ostree_diff_dirs_with_options(), expose via cmdline 2017-03-21 13:38:04 +00:00
ostree-dummy-enumtypes.c build: Don't run glib-mkenums if there's nothing to do 2016-05-12 17:15:50 +00:00
ostree-dummy-enumtypes.h build: Don't run glib-mkenums if there's nothing to do 2016-05-12 17:15:50 +00:00
ostree-enumtypes.c.template Move the include directive to the enum template 2017-07-17 15:14:44 +00:00
ostree-enumtypes.h.template Make enum generation private by default, export stub symbol 2016-05-12 17:15:50 +00:00
ostree-fetcher-curl.c lib/pull: Avoid journaling 404s for optional content 2017-07-13 21:41:00 +00:00
ostree-fetcher-soup.c Update libglnx, port various bits to new API 2017-07-18 19:18:38 +00:00
ostree-fetcher-uri.c libcurl backend 2017-02-09 16:37:45 +00:00
ostree-fetcher-util.c lib/pull: Avoid journaling 404s for optional content 2017-07-13 21:41:00 +00:00
ostree-fetcher-util.h lib/pull: Avoid journaling 404s for optional content 2017-07-13 21:41:00 +00:00
ostree-fetcher.h lib/pull: Avoid journaling 404s for optional content 2017-07-13 21:41:00 +00:00
ostree-gpg-verifier.c Update libglnx, port some uses to newer APIs 2017-07-24 18:43:57 +00:00
ostree-gpg-verifier.h Add "gpgkeypath" option to remotes 2016-11-17 09:44:07 +00:00
ostree-gpg-verify-result-private.h libotutil: Establish a place for GPG utilities 2015-05-01 10:20:34 -04:00
ostree-gpg-verify-result.c tree-wide: Add+run spatch to use glnx_throw() 2017-05-26 19:27:11 +00:00
ostree-gpg-verify-result.h repo: Factor out the check of gpg result to a separate function 2016-05-27 11:20:00 +00:00
ostree-impl-system-generator.c Update libglnx, port various bits to new API 2017-07-18 19:18:38 +00:00
ostree-kernel-args.c Filter bootloader supplied kernel cmdline options 2016-11-01 18:00:37 +00:00
ostree-kernel-args.h Filter bootloader supplied kernel cmdline options 2016-11-01 18:00:37 +00:00
ostree-libarchive-input-stream.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-libarchive-input-stream.h Define and use cleanup helpers for libarchive 2016-12-05 15:20:56 +00:00
ostree-libarchive-private.h libostree: added empty ot_cleanup_{read,write}_archive macros. 2017-02-10 14:06:37 +00:00
ostree-linuxfsutil.c lib/fsutil: Port to new code style 2017-05-26 19:05:53 +00:00
ostree-linuxfsutil.h lib/fsutil: Delete unused GFile ioctl method 2017-05-26 19:05:53 +00:00
ostree-lzma-common.c static-delta: Handle LZMA_BUF_ERROR returned by zlib 2015-10-10 10:27:17 -04:00
ostree-lzma-common.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-lzma-compressor.c docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-lzma-compressor.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-lzma-decompressor.c docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-lzma-decompressor.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-metalink.c lib/pull: Avoid journaling 404s for optional content 2017-07-13 21:41:00 +00:00
ostree-metalink.h fetcher: Define an abstraction over SoupURI 2016-12-07 23:00:58 +00:00
ostree-mutable-tree.c codebase: start using GLNX_HASH_TABLE_FOREACH macros 2017-06-28 16:37:15 +00:00
ostree-mutable-tree.h docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-ref.c lib/ref: Add OstreeCollectionRef type for globally unique refs 2017-06-26 15:56:07 +00:00
ostree-ref.h lib/ref: Suppress more collection ref methods from introspection 2017-06-27 11:56:44 +00:00
ostree-remote-private.h lib/remote: Add arguments to internal OstreeRemote constructor 2017-05-19 15:01:59 +00:00
ostree-remote.c lib/remote: Add arguments to internal OstreeRemote constructor 2017-05-19 15:01:59 +00:00
ostree-remote.h lib/remote: Fix ‘Since’ line for OstreeRemote 2017-06-21 13:00:55 +00:00
ostree-repo-checkout.c Update libglnx, port various bits to new API 2017-07-18 19:18:38 +00:00
ostree-repo-commit.c Update libglnx, port some uses to newer APIs 2017-07-24 18:43:57 +00:00
ostree-repo-deprecated.h libostree: mark ostree_repo_checkout_tree_at as deprecated 2016-07-30 11:24:52 +00:00
ostree-repo-file-enumerator.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-repo-file-enumerator.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-repo-file.c lib/repofile: Follow symlinks for `g_file_read()` 2017-06-07 20:54:32 +00:00
ostree-repo-file.h lib: Remove ostree_repo_file_make_empty_tree 2016-04-17 13:52:07 +00:00
ostree-repo-finder-avahi-parser.c lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder-avahi-private.h lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder-avahi.c lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder-avahi.h lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder-config.c lib/repo-finder: Add config-file based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder-config.h lib/repo-finder: Add config-file based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder-mount.c lib/repo-finder: Add mount based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder-mount.h lib/repo-finder: Add mount based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder.c lib/repo-finder: Add basic support for finding remote URIs by ref name 2017-06-26 15:56:07 +00:00
ostree-repo-finder.h lib/repo-finder: Add basic support for finding remote URIs by ref name 2017-06-26 15:56:07 +00:00
ostree-repo-libarchive.c lib: Add a helper to convert struct stat → GFileInfo 2017-06-29 18:17:28 +00:00
ostree-repo-private.h lib/core: Add #defines for ref/collection binding 2017-07-26 18:52:04 +00:00
ostree-repo-prune.c codebase: start using GLNX_HASH_TABLE_FOREACH macros 2017-06-28 16:37:15 +00:00
ostree-repo-pull.c lib/core: Add #defines for ref/collection binding 2017-07-26 18:52:04 +00:00
ostree-repo-refs.c lib/refs: Add runtime error checking for collection ID validity 2017-06-26 15:56:07 +00:00
ostree-repo-static-delta-compilation-analysis.c [ASAN] lib: Squash various leaks in library and commandline 2016-11-21 16:34:06 +00:00
ostree-repo-static-delta-compilation.c libutil: Add a helper for O_TMPFILE + mmap() 2017-06-29 14:46:18 +00:00
ostree-repo-static-delta-core.c lib/deltas: More porting to new code style 2017-06-29 14:46:18 +00:00
ostree-repo-static-delta-private.h lib: Always checksum content in deltas 2016-12-06 15:59:35 +00:00
ostree-repo-static-delta-processing.c Update libglnx, port various bits to new API 2017-07-18 19:18:38 +00:00
ostree-repo-traverse.c traverse: Use g_hash_table_add 2016-11-30 18:51:26 +00:00
ostree-repo.c Update libglnx, port some uses to newer APIs 2017-07-24 18:43:57 +00:00
ostree-repo.h lib/repo: Add OSTREE_REPO_METADATA_REF as a well-known metadata store 2017-07-11 19:50:32 +00:00
ostree-rollsum.c rollsum: Fix assertion for CRC matches 2015-08-25 09:26:22 -04:00
ostree-rollsum.h lib/deltas: Port to more to new code style 2017-06-26 14:42:56 +00:00
ostree-sepolicy-private.h checkout: Add SELinux labeling for checkout, use in deploy 2017-04-25 16:52:33 +00:00
ostree-sepolicy.c libglnx: bump and use new helper methods 2017-05-12 21:02:16 +00:00
ostree-sepolicy.h sepolicy: Add ostree_sepolicy_new_at() 2017-03-22 16:24:06 +00:00
ostree-soup-form.c libcurl backend 2017-02-09 16:37:45 +00:00
ostree-soup-uri.c libcurl backend 2017-02-09 16:37:45 +00:00
ostree-soup-uri.h libcurl backend 2017-02-09 16:37:45 +00:00
ostree-sysroot-cleanup.c lib/sysroot: Add some g_prefix_error() for ostree_sysroot_cleanup() 2017-06-15 00:08:29 +00:00
ostree-sysroot-deploy.c Update libglnx, port some uses to newer APIs 2017-07-24 18:43:57 +00:00
ostree-sysroot-private.h Add a notion of "physical" sysroot, use for remote writing 2017-07-18 18:58:06 +00:00
ostree-sysroot-upgrader.c lib: Add #defines for current well-known metadata keys 2017-07-21 14:53:33 +00:00
ostree-sysroot-upgrader.h upgrade: Add support for --pull-only and --deploy-only 2017-02-27 20:57:23 +00:00
ostree-sysroot.c Add a notion of "physical" sysroot, use for remote writing 2017-07-18 18:58:06 +00:00
ostree-sysroot.h lib/sysroot: Add API to get pending/rollback for given stateroot 2017-06-14 09:56:01 +00:00
ostree-tls-cert-interaction.c fetcher: Rework API to use strings for tls keys/db 2017-01-19 10:47:15 +00:00
ostree-tls-cert-interaction.h fetcher: Rework API to use strings for tls keys/db 2017-01-19 10:47:15 +00:00
ostree-types.h libostree: Make OstreeRemote a public and internal API 2017-05-08 18:48:07 +00:00
ostree-varint.c syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-varint.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-version.h.in ostreee-version.h.in: Added Since: version annotations 2017-06-15 12:59:59 +00:00
ostree.h lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00

README.md

Repository design

At the heart of OSTree is the repository. It's very similar to git, with the idea of content-addressed storage. However, OSTree is designed to store operating system binaries, not source code. There are several consequences to this. The key difference as compared to git is that the OSTree definition of "content" includes key Unix metadata such as owner uid/gid, as well as all extended attributes.

Essentially OSTree is designed so that if two files have the same OSTree checksum, it's safe to replace them with a hard link. This fundamental design means that an OSTree repository imposes negligible overhead. In contrast, a git repository stores copies of zlib-compressed data.

Key differences versus git

  • As mentioned above, extended attributes and owner uid/gid are versioned
  • Optimized for Unix hardlinks between repository and checkout
  • SHA256 instead of SHA1
  • Support for empty directories

Binary files

While this is still in planning, I plan to heavily optimize OSTree for versioning ELF operating systems. In industry jargon, this would be "content-aware storage".

Trimming history

OSTree will also be optimized to trim intermediate history; in theory one can regenerate binaries from corresponding (git) source code, so we don't need to keep all possible builds over time.

MILESTONE 1

  • Basic pack files (like git)

MILESTONE 2

  • Store checksums as ay
  • Drop version/metadata from tree/dirmeta objects
  • Add index size to superindex, pack size to index
    • So pull can calculate how much we need to download
  • Split pack files into metadata/data
  • pull: Extract all we can from each packfile one at a time, then delete it
  • Restructure repository so that links can be generated as a cache; i.e. objects/raw, pack files are now the canonical
  • For files, checksum combination of metadata variant + raw data
    • i.e. there is only OSTREE_OBJECT_TYPE_FILE (again)

MILESTONE 3

  • Drop archive/raw distinction - archive repositories always generate packfiles per commit
  • Include git packv4 ideas:
    • metadata packfiles have string dictionary (tree filenames and checksums)
    • data packfiles match up similar objects
  • Rolling checksums for partitioning large files? Kernel debuginfo
  • Improved pack clustering
    • file fingerprinting?
  • ELF-x86 aware deltas

Related work in storage

git: http://git-scm.com/ Venti: http://plan9.bell-labs.com/magic/man2html/6/venti Elephant FS: http://www.hpl.hp.com/personal/Alistair_Veitch/papers/elephant-hotos/index.html

Compression

xdelta: http://xdelta.org/ Bsdiff: http://www.daemonology.net/bsdiff/ xz: http://tukaani.org/xz/