packaging
/
ostree
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ostree/src/libostree
History
Colin Walters 88b25dc65c tree-wide: Replace archive-z2 with archive 
In almost all places. There are just a few exceptions; one tricky bit for
example is that the repo config must still have `mode=archive-z2`, since
`archive` used to mean something else. (We could very likely just get rid of
that check, but eh, later).

I also added a test that one can still do `ostree repo init --mode=archive-z2`.

Closes: #1125
Approved by: jlebon
 		2017-09-01 20:54:12 +00:00
..
README-gpg pull: Verify commits with gpg signatures from detached metadata 2013-09-29 14:49:47 -04:00
README.md core: Associate branches with remotes, move trigger runs into checkout 2012-04-03 23:46:34 -04:00
bupsplit.c lib: Move the bupsplit selftest into our test framework 2017-02-02 16:51:36 +00:00
bupsplit.h lib: Move the bupsplit selftest into our test framework 2017-02-02 16:51:36 +00:00
libostree-devel.sym build-sys: Post-release version bump 2017-08-17 18:18:21 +00:00
libostree-experimental.sym lib/repo-refs: Add ostree_repo_remote_list_collection_refs() API 2017-08-08 13:59:58 +00:00
libostree-released.sym Release 2017.10 2017-08-17 18:18:21 +00:00
ostree-1.pc.in libostree: Expose $OSTREE_FEATURES in the pkg-config file 2017-05-08 18:48:07 +00:00
ostree-async-progress.c libostree: Fix potential use of uninitialised memory in progress API 2017-05-05 14:58:03 +00:00
ostree-async-progress.h libostree: Add multiple getter/setter support to OstreeAsyncProgress 2017-04-29 11:50:15 +00:00
ostree-autocleanups.h lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-bloom-private.h lib/bloom: Add an internal bloom filter implementation 2017-06-26 15:56:07 +00:00
ostree-bloom.c lib/bloom: Add an internal bloom filter implementation 2017-06-26 15:56:07 +00:00
ostree-bootconfig-parser.c codebase: start using GLNX_HASH_TABLE_FOREACH macros 2017-06-28 16:37:15 +00:00
ostree-bootconfig-parser.h lib: Introduce versioned symbols 2016-03-01 21:45:26 -05:00
ostree-bootloader-grub2.c grub2: Handle aarch64 (at least on Fedora/RHEL) 2017-07-27 18:21:00 +00:00
ostree-bootloader-grub2.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-bootloader-syslinux.c libglnx porting: Port "load utf8 file" contents API callers 2016-05-30 11:33:28 +00:00
ostree-bootloader-syslinux.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-bootloader-uboot.c u-boot: Merge ostree's and systems uEnv.txt 2016-08-27 13:11:22 +00:00
ostree-bootloader-uboot.h libostree: Make OstreeBootloader private 2013-09-15 20:26:13 -04:00
ostree-bootloader.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-bootloader.h tree-wide: Fix the build with old glib (Ubuntu Trusty etc.) 2017-08-03 16:23:41 +00:00
ostree-chain-input-stream.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-chain-input-stream.h lib: Introduce versioned symbols 2016-03-01 21:45:26 -05:00
ostree-checksum-input-stream.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-checksum-input-stream.h lib: Introduce versioned symbols 2016-03-01 21:45:26 -05:00
ostree-cmdprivate.c Switch to using a systemd generator for /var 2017-05-16 16:13:05 +00:00
ostree-cmdprivate.h Switch to using a systemd generator for /var 2017-05-16 16:13:05 +00:00
ostree-core-private.h lib/pull: Add support for timestamp-check option, use in upgrader 2017-08-21 18:24:07 +00:00
ostree-core.c tree-wide: Replace archive-z2 with archive 2017-09-01 20:54:12 +00:00
ostree-core.h lib/core: Add #defines for ref/collection binding 2017-07-26 18:52:04 +00:00
ostree-deployment-private.h docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-deployment.c ostree-deployment.c: simplify equality check 2017-08-16 13:17:14 +00:00
ostree-deployment.h admin: Add an `unlock` command, and libostree API 2016-03-23 11:09:09 -04:00
ostree-diff.c diff: Port some to new code style 2017-05-09 18:33:12 +00:00
ostree-diff.h diff: Add ostree_diff_dirs_with_options(), expose via cmdline 2017-03-21 13:38:04 +00:00
ostree-dummy-enumtypes.c build: Don't run glib-mkenums if there's nothing to do 2016-05-12 17:15:50 +00:00
ostree-dummy-enumtypes.h build: Don't run glib-mkenums if there's nothing to do 2016-05-12 17:15:50 +00:00
ostree-enumtypes.c.template Move the include directive to the enum template 2017-07-17 15:14:44 +00:00
ostree-enumtypes.h.template Make enum generation private by default, export stub symbol 2016-05-12 17:15:50 +00:00
ostree-fetcher-curl.c lib/curl: Only check individual request errors 2017-08-02 15:44:00 +00:00
ostree-fetcher-soup.c tree-wide: Use g_autoptr(Ostree*) 2017-08-03 13:48:12 +00:00
ostree-fetcher-uri.c libcurl backend 2017-02-09 16:37:45 +00:00
ostree-fetcher-util.c lib/pull: Avoid journaling 404s for optional content 2017-07-13 21:41:00 +00:00
ostree-fetcher-util.h lib/pull: Avoid journaling 404s for optional content 2017-07-13 21:41:00 +00:00
ostree-fetcher.h lib/pull: Avoid journaling 404s for optional content 2017-07-13 21:41:00 +00:00
ostree-gpg-verifier.c lib: Port gpg verification for remotes to fd-relative 2017-08-08 15:50:25 +00:00
ostree-gpg-verifier.h lib: Port gpg verification for remotes to fd-relative 2017-08-08 15:50:25 +00:00
ostree-gpg-verify-result-private.h libotutil: Establish a place for GPG utilities 2015-05-01 10:20:34 -04:00
ostree-gpg-verify-result.c lib/gpg-verify: Add an OstreeGpgError error domain 2017-08-10 13:38:40 +00:00
ostree-gpg-verify-result.h lib/gpg-verify: Add an OstreeGpgError error domain 2017-08-10 13:38:40 +00:00
ostree-impl-system-generator.c tree-wide: Remove trailing semicolon from autoptr declarations 2017-08-07 17:42:32 +00:00
ostree-kernel-args.c Filter bootloader supplied kernel cmdline options 2016-11-01 18:00:37 +00:00
ostree-kernel-args.h Filter bootloader supplied kernel cmdline options 2016-11-01 18:00:37 +00:00
ostree-libarchive-input-stream.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-libarchive-input-stream.h Define and use cleanup helpers for libarchive 2016-12-05 15:20:56 +00:00
ostree-libarchive-private.h libarchive: Add support for translating paths during commit 2017-08-30 14:30:30 +00:00
ostree-linuxfsutil.c lib/fsutil: Port to new code style 2017-05-26 19:05:53 +00:00
ostree-linuxfsutil.h lib/fsutil: Delete unused GFile ioctl method 2017-05-26 19:05:53 +00:00
ostree-lzma-common.c static-delta: Handle LZMA_BUF_ERROR returned by zlib 2015-10-10 10:27:17 -04:00
ostree-lzma-common.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-lzma-compressor.c docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-lzma-compressor.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-lzma-decompressor.c docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-lzma-decompressor.h deltas: Link to liblzma, add internal API to use it 2014-04-29 10:59:57 -04:00
ostree-metalink.c lib/pull: Avoid journaling 404s for optional content 2017-07-13 21:41:00 +00:00
ostree-metalink.h tree-wide: Use g_autoptr(Ostree*) 2017-08-03 13:48:12 +00:00
ostree-mutable-tree.c tree-wide: Fix the build with old glib (Ubuntu Trusty etc.) 2017-08-03 16:23:41 +00:00
ostree-mutable-tree.h docs: Get API docs working again 2016-06-09 18:15:49 +00:00
ostree-ref.c lib/ref: Add OstreeCollectionRef type for globally unique refs 2017-06-26 15:56:07 +00:00
ostree-ref.h lib/ref: Suppress more collection ref methods from introspection 2017-06-27 11:56:44 +00:00
ostree-remote-private.h lib/remote: Add arguments to internal OstreeRemote constructor 2017-05-19 15:01:59 +00:00
ostree-remote.c lib/remote: Add arguments to internal OstreeRemote constructor 2017-05-19 15:01:59 +00:00
ostree-remote.h lib/remote: Fix ‘Since’ line for OstreeRemote 2017-06-21 13:00:55 +00:00
ostree-repo-checkout.c checkout: add an extra checkout_overwrite mode 2017-09-01 15:42:50 +00:00
ostree-repo-commit.c commit: filter out selinux label before commit 2017-08-31 12:07:46 +00:00
ostree-repo-deprecated.h libostree: mark ostree_repo_checkout_tree_at as deprecated 2016-07-30 11:24:52 +00:00
ostree-repo-file-enumerator.c Remove unnecessary #include "libgsystem.h" 2015-05-06 22:07:11 -04:00
ostree-repo-file-enumerator.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-repo-file.c lib/repofile: Follow symlinks for `g_file_read()` 2017-06-07 20:54:32 +00:00
ostree-repo-file.h lib: Remove ostree_repo_file_make_empty_tree 2016-04-17 13:52:07 +00:00
ostree-repo-finder-avahi-parser.c lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder-avahi-private.h lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder-avahi.c lib/repo-finder: Emit gpg-verify-summary=false in dynamic remote config 2017-08-08 18:08:05 +00:00
ostree-repo-finder-avahi.h lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder-config.c lib/repo-finder-config: Add some more debug output 2017-08-11 18:07:29 +00:00
ostree-repo-finder-config.h lib/repo-finder: Add config-file based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder-mount.c lib/repo-refs: Include remote refs when using collections 2017-08-24 19:57:33 +00:00
ostree-repo-finder-mount.h lib/repo-finder: Add mount based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00
ostree-repo-finder.c lib/repo-finder: Avoid a potential unref-of-NULL crash 2017-08-08 13:59:58 +00:00
ostree-repo-finder.h lib/repo-finder: Add basic support for finding remote URIs by ref name 2017-06-26 15:56:07 +00:00
ostree-repo-libarchive.c libarchive: Add support for translating paths during commit 2017-08-30 14:30:30 +00:00
ostree-repo-private.h lib/repo-refs: Include remote refs when using collections 2017-08-24 19:57:33 +00:00
ostree-repo-prune.c lib/repo-refs: Include remote refs when using collections 2017-08-24 19:57:33 +00:00
ostree-repo-pull.c tree-wide: Replace archive-z2 with archive 2017-09-01 20:54:12 +00:00
ostree-repo-refs.c lib/repo-refs: Fix typos in last commit 2017-08-25 16:54:15 +00:00
ostree-repo-static-delta-compilation-analysis.c [ASAN] lib: Squash various leaks in library and commandline 2016-11-21 16:34:06 +00:00
ostree-repo-static-delta-compilation.c libutil: Add a helper for O_TMPFILE + mmap() 2017-06-29 14:46:18 +00:00
ostree-repo-static-delta-core.c lib/deltas: Squash Coverity warning for div-by-zero in delta show 2017-08-07 16:10:08 +00:00
ostree-repo-static-delta-private.h lib: Always checksum content in deltas 2016-12-06 15:59:35 +00:00
ostree-repo-static-delta-processing.c tree-wide: Replace archive-z2 with archive 2017-09-01 20:54:12 +00:00
ostree-repo-traverse.c traverse: Use g_hash_table_add 2016-11-30 18:51:26 +00:00
ostree-repo.c lib/repo-refs: Include remote refs when using collections 2017-08-24 19:57:33 +00:00
ostree-repo.h checkout: add an extra checkout_overwrite mode 2017-09-01 15:42:50 +00:00
ostree-rollsum.c rollsum: Fix assertion for CRC matches 2015-08-25 09:26:22 -04:00
ostree-rollsum.h lib/deltas: Port to more to new code style 2017-06-26 14:42:56 +00:00
ostree-sepolicy-private.h checkout: Add SELinux labeling for checkout, use in deploy 2017-04-25 16:52:33 +00:00
ostree-sepolicy.c lib/sepolicy: Drop duplicate assignment 2017-08-08 14:11:07 +00:00
ostree-sepolicy.h sepolicy: Add ostree_sepolicy_new_at() 2017-03-22 16:24:06 +00:00
ostree-soup-form.c libcurl backend 2017-02-09 16:37:45 +00:00
ostree-soup-uri.c libcurl backend 2017-02-09 16:37:45 +00:00
ostree-soup-uri.h libcurl backend 2017-02-09 16:37:45 +00:00
ostree-sysroot-cleanup.c lib/sysroot: Add some g_prefix_error() for ostree_sysroot_cleanup() 2017-06-15 00:08:29 +00:00
ostree-sysroot-deploy.c lib/deploy: Add .img to end of initramfs in /usr/lib/modules 2017-08-21 16:41:50 +00:00
ostree-sysroot-private.h repo: Introduce ostree_repo_open_at() and ostree_repo_create_at() 2017-08-15 12:35:10 +00:00
ostree-sysroot-upgrader.c lib/pull: Add support for timestamp-check option, use in upgrader 2017-08-21 18:24:07 +00:00
ostree-sysroot-upgrader.h upgrade: Add support for --pull-only and --deploy-only 2017-02-27 20:57:23 +00:00
ostree-sysroot.c ostree-sysroot: make simple_write_deployment smarter 2017-08-25 01:02:15 +00:00
ostree-sysroot.h ostree-sysroot: make simple_write_deployment smarter 2017-08-25 01:02:15 +00:00
ostree-tls-cert-interaction.c fetcher: Rework API to use strings for tls keys/db 2017-01-19 10:47:15 +00:00
ostree-tls-cert-interaction.h tree-wide: Fix the build with old glib (Ubuntu Trusty etc.) 2017-08-03 16:23:41 +00:00
ostree-types.h libostree: Make OstreeRemote a public and internal API 2017-05-08 18:48:07 +00:00
ostree-varint.c syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-varint.h syntax-check: Remove empty lines at the end of file 2015-02-02 15:07:56 +01:00
ostree-version.h.in ostreee-version.h.in: Added Since: version annotations 2017-06-15 12:59:59 +00:00
ostree.h lib/repo-finder: Add Avahi based OstreeRepoFinder implementation 2017-06-26 15:56:07 +00:00

README.md

Repository design

At the heart of OSTree is the repository. It's very similar to git, with the idea of content-addressed storage. However, OSTree is designed to store operating system binaries, not source code. There are several consequences to this. The key difference as compared to git is that the OSTree definition of "content" includes key Unix metadata such as owner uid/gid, as well as all extended attributes.

Essentially OSTree is designed so that if two files have the same OSTree checksum, it's safe to replace them with a hard link. This fundamental design means that an OSTree repository imposes negligible overhead. In contrast, a git repository stores copies of zlib-compressed data.

Key differences versus git

  • As mentioned above, extended attributes and owner uid/gid are versioned
  • Optimized for Unix hardlinks between repository and checkout
  • SHA256 instead of SHA1
  • Support for empty directories

Binary files

While this is still in planning, I plan to heavily optimize OSTree for versioning ELF operating systems. In industry jargon, this would be "content-aware storage".

Trimming history

OSTree will also be optimized to trim intermediate history; in theory one can regenerate binaries from corresponding (git) source code, so we don't need to keep all possible builds over time.

MILESTONE 1

  • Basic pack files (like git)

MILESTONE 2

  • Store checksums as ay
  • Drop version/metadata from tree/dirmeta objects
  • Add index size to superindex, pack size to index
    • So pull can calculate how much we need to download
  • Split pack files into metadata/data
  • pull: Extract all we can from each packfile one at a time, then delete it
  • Restructure repository so that links can be generated as a cache; i.e. objects/raw, pack files are now the canonical
  • For files, checksum combination of metadata variant + raw data
    • i.e. there is only OSTREE_OBJECT_TYPE_FILE (again)

MILESTONE 3

  • Drop archive/raw distinction - archive repositories always generate packfiles per commit
  • Include git packv4 ideas:
    • metadata packfiles have string dictionary (tree filenames and checksums)
    • data packfiles match up similar objects
  • Rolling checksums for partitioning large files? Kernel debuginfo
  • Improved pack clustering
    • file fingerprinting?
  • ELF-x86 aware deltas

Related work in storage

git: http://git-scm.com/ Venti: http://plan9.bell-labs.com/magic/man2html/6/venti Elephant FS: http://www.hpl.hp.com/personal/Alistair_Veitch/papers/elephant-hotos/index.html

Compression

xdelta: http://xdelta.org/ Bsdiff: http://www.daemonology.net/bsdiff/ xz: http://tukaani.org/xz/