8e6e64a5ad
Previously we were doing e.g. `ot_util_filename_validate()` specifically inline in dirtree objects, but only *after* writing them into the staging directory (by default). In (non-default) cases such as not using a transaction, such an object could be written directly into the repo. A notable gap here is that `pull-local --untrusted` was *not* doing this verification, just checksums. We harden that (and also the static delta writing path, really *everything* that calls `ostree_repo_write_metadata()` to also do "structure" validation which includes path traversal checks. Basically, let's try hard to avoid having badly structured objects even in the repo. One thing that sucks in this patch is that we need to allocate a "bounce buffer" for metadata in the static delta path, because GVariant imposes alignment requirements, which I screwed up and didn't fulfill when designing deltas. It actually didn't matter before because we weren't parsing them, but now we are. In theory we could check alignment but ...eh, not worth it, at least not until we change the delta compiler to emit aligned metadata which actually may be quite tricky. (Big picture I doubt this really matters much right now but I'm not going to pull out a profiler yet for this) The pull test was extended to check we didn't even write a dirtree with path traversal into the staging directory. There's a bit of code motion in extracting `_ostree_validate_structureof_metadata()` from `fsck_metadata_object()`. Then `_ostree_verify_metadata_object()` builds on that to do checksum verification too. Closes: #1412 Approved by: jlebon |
||
|---|---|---|
| .. | ||
| README-gpg | ||
| README.md | ||
| bupsplit.c | ||
| bupsplit.h | ||
| libostree-devel.sym | ||
| libostree-experimental.sym | ||
| libostree-released.sym | ||
| ostree-1.pc.in | ||
| ostree-async-progress.c | ||
| ostree-async-progress.h | ||
| ostree-autocleanups.h | ||
| ostree-bloom-private.h | ||
| ostree-bloom.c | ||
| ostree-bootconfig-parser.c | ||
| ostree-bootconfig-parser.h | ||
| ostree-bootloader-grub2.c | ||
| ostree-bootloader-grub2.h | ||
| ostree-bootloader-syslinux.c | ||
| ostree-bootloader-syslinux.h | ||
| ostree-bootloader-uboot.c | ||
| ostree-bootloader-uboot.h | ||
| ostree-bootloader.c | ||
| ostree-bootloader.h | ||
| ostree-chain-input-stream.c | ||
| ostree-chain-input-stream.h | ||
| ostree-checksum-input-stream.c | ||
| ostree-checksum-input-stream.h | ||
| ostree-cmdprivate.c | ||
| ostree-cmdprivate.h | ||
| ostree-core-private.h | ||
| ostree-core.c | ||
| ostree-core.h | ||
| ostree-deployment-private.h | ||
| ostree-deployment.c | ||
| ostree-deployment.h | ||
| ostree-diff.c | ||
| ostree-diff.h | ||
| ostree-dummy-enumtypes.c | ||
| ostree-dummy-enumtypes.h | ||
| ostree-enumtypes.c.template | ||
| ostree-enumtypes.h.template | ||
| ostree-fetcher-curl.c | ||
| ostree-fetcher-soup.c | ||
| ostree-fetcher-uri.c | ||
| ostree-fetcher-util.c | ||
| ostree-fetcher-util.h | ||
| ostree-fetcher.h | ||
| ostree-gpg-verifier.c | ||
| ostree-gpg-verifier.h | ||
| ostree-gpg-verify-result-private.h | ||
| ostree-gpg-verify-result.c | ||
| ostree-gpg-verify-result.h | ||
| ostree-impl-system-generator.c | ||
| ostree-kernel-args.c | ||
| ostree-kernel-args.h | ||
| ostree-libarchive-input-stream.c | ||
| ostree-libarchive-input-stream.h | ||
| ostree-libarchive-private.h | ||
| ostree-linuxfsutil.c | ||
| ostree-linuxfsutil.h | ||
| ostree-lzma-common.c | ||
| ostree-lzma-common.h | ||
| ostree-lzma-compressor.c | ||
| ostree-lzma-compressor.h | ||
| ostree-lzma-decompressor.c | ||
| ostree-lzma-decompressor.h | ||
| ostree-metalink.c | ||
| ostree-metalink.h | ||
| ostree-mutable-tree.c | ||
| ostree-mutable-tree.h | ||
| ostree-ref.c | ||
| ostree-ref.h | ||
| ostree-remote-private.h | ||
| ostree-remote.c | ||
| ostree-remote.h | ||
| ostree-repo-checkout.c | ||
| ostree-repo-commit.c | ||
| ostree-repo-deprecated.h | ||
| ostree-repo-file-enumerator.c | ||
| ostree-repo-file-enumerator.h | ||
| ostree-repo-file.c | ||
| ostree-repo-file.h | ||
| ostree-repo-finder-avahi-parser.c | ||
| ostree-repo-finder-avahi-private.h | ||
| ostree-repo-finder-avahi.c | ||
| ostree-repo-finder-avahi.h | ||
| ostree-repo-finder-config.c | ||
| ostree-repo-finder-config.h | ||
| ostree-repo-finder-mount.c | ||
| ostree-repo-finder-mount.h | ||
| ostree-repo-finder-override.c | ||
| ostree-repo-finder-override.h | ||
| ostree-repo-finder.c | ||
| ostree-repo-finder.h | ||
| ostree-repo-libarchive.c | ||
| ostree-repo-private.h | ||
| ostree-repo-prune.c | ||
| ostree-repo-pull-private.h | ||
| ostree-repo-pull.c | ||
| ostree-repo-refs.c | ||
| ostree-repo-static-delta-compilation-analysis.c | ||
| ostree-repo-static-delta-compilation.c | ||
| ostree-repo-static-delta-core.c | ||
| ostree-repo-static-delta-private.h | ||
| ostree-repo-static-delta-processing.c | ||
| ostree-repo-traverse.c | ||
| ostree-repo.c | ||
| ostree-repo.h | ||
| ostree-rollsum.c | ||
| ostree-rollsum.h | ||
| ostree-sepolicy-private.h | ||
| ostree-sepolicy.c | ||
| ostree-sepolicy.h | ||
| ostree-soup-form.c | ||
| ostree-soup-uri.c | ||
| ostree-soup-uri.h | ||
| ostree-sysroot-cleanup.c | ||
| ostree-sysroot-deploy.c | ||
| ostree-sysroot-private.h | ||
| ostree-sysroot-upgrader.c | ||
| ostree-sysroot-upgrader.h | ||
| ostree-sysroot.c | ||
| ostree-sysroot.h | ||
| ostree-tls-cert-interaction.c | ||
| ostree-tls-cert-interaction.h | ||
| ostree-types.h | ||
| ostree-varint.c | ||
| ostree-varint.h | ||
| ostree-version.h.in | ||
| ostree.h | ||
README.md
Repository design
At the heart of OSTree is the repository. It's very similar to git, with the idea of content-addressed storage. However, OSTree is designed to store operating system binaries, not source code. There are several consequences to this. The key difference as compared to git is that the OSTree definition of "content" includes key Unix metadata such as owner uid/gid, as well as all extended attributes.
Essentially OSTree is designed so that if two files have the same OSTree checksum, it's safe to replace them with a hard link. This fundamental design means that an OSTree repository imposes negligible overhead. In contrast, a git repository stores copies of zlib-compressed data.
Key differences versus git
- As mentioned above, extended attributes and owner uid/gid are versioned
- Optimized for Unix hardlinks between repository and checkout
- SHA256 instead of SHA1
- Support for empty directories
Binary files
While this is still in planning, I plan to heavily optimize OSTree for versioning ELF operating systems. In industry jargon, this would be "content-aware storage".
Trimming history
OSTree will also be optimized to trim intermediate history; in theory one can regenerate binaries from corresponding (git) source code, so we don't need to keep all possible builds over time.
MILESTONE 1
- Basic pack files (like git)
MILESTONE 2
- Store checksums as ay
- Drop version/metadata from tree/dirmeta objects
- Add index size to superindex, pack size to index
- So pull can calculate how much we need to download
- Split pack files into metadata/data
- pull: Extract all we can from each packfile one at a time, then delete it
- Restructure repository so that links can be generated as a cache; i.e. objects/raw, pack files are now the canonical
- For files, checksum combination of metadata variant + raw data
- i.e. there is only OSTREE_OBJECT_TYPE_FILE (again)
MILESTONE 3
- Drop archive/raw distinction - archive repositories always generate packfiles per commit
- Include git packv4 ideas:
- metadata packfiles have string dictionary (tree filenames and checksums)
- data packfiles match up similar objects
- Rolling checksums for partitioning large files? Kernel debuginfo
- Improved pack clustering
- file fingerprinting?
- ELF-x86 aware deltas
Related work in storage
git: http://git-scm.com/ Venti: http://plan9.bell-labs.com/magic/man2html/6/venti Elephant FS: http://www.hpl.hp.com/personal/Alistair_Veitch/papers/elephant-hotos/index.html
Compression
xdelta: http://xdelta.org/ Bsdiff: http://www.daemonology.net/bsdiff/ xz: http://tukaani.org/xz/