Matthew Leeds
daa57b4630
Normally, a configured remote will only serve refs with one associated collection ID, but temporary remotes such as USB drives or LAN peers can serve refs from multiple collection IDs which may use different GPG keyrings. So the OstreeRepoFinderMount and OstreeRepoFinderAvahi classes create dynamic OstreeRemote objects for each (uri, keyring) pair. So if for example the USB mounted at /mnt/usb serves content from the configured remotes "eos-apps" and "eos-sdk", the OstreeRepoFinderResult array returned by ostree_repo_find_remotes_async() will have one result with a remote called something like file_mnt_usb_eos-apps.trustedkeys.gpg and the list of refs on the USB that came from eos-apps, and another result with a remote file_mnt_usb_eos-sdk.trustedkeys.gpg and the list of refs from eos-sdk. Unfortunately while OstreeRepoFinderMount and OstreeRepoFinderAvahi correctly only include refs in a result if the ref uses the associated keyring, the find_remotes_cb() function used to clean up the set of results looks at the remote summary file and includes every ref that's in the intersection with the requested refs, regardless of whether it uses a different remote's keyring. This leads to an error when you try to pull from a USB containing refs from different collection IDs: the pull using the wrong collection ID will error out with "Refspec not found" and the result with the correct keyring will then be ignored "as it has no relevant refs or they have already been pulled." So the pull ultimately fails. This commit fixes the issue by filtering refs coming from a dynamic remote, so that only ones with the collection ID associated with the keyring remote are examined. This only needs to be done for dynamic remotes because you should be able to pull any ref from a configured remote using its keyring. It's also only done when looking at the collection map in the summary file, because LAN/USB remotes won't have a "main" collection ID set (OSTREE_SUMMARY_COLLECTION_ID). Closes: #1695 Approved by: pwithnall
Repository design
At the heart of OSTree is the repository. It's very similar to git, with the idea of content-addressed storage. However, OSTree is designed to store operating system binaries, not source code. There are several consequences to this. The key difference as compared to git is that the OSTree definition of "content" includes key Unix metadata such as owner uid/gid, as well as all extended attributes.
Essentially OSTree is designed so that if two files have the same OSTree checksum, it's safe to replace them with a hard link. This fundamental design means that an OSTree repository imposes negligible overhead. In contrast, a git repository stores copies of zlib-compressed data.
Key differences versus git
- As mentioned above, extended attributes and owner uid/gid are versioned
- Optimized for Unix hardlinks between repository and checkout
- SHA256 instead of SHA1
- Support for empty directories
Binary files
While this is still in planning, I plan to heavily optimize OSTree for versioning ELF operating systems. In industry jargon, this would be "content-aware storage".
Trimming history
OSTree will also be optimized to trim intermediate history; in theory one can regenerate binaries from corresponding (git) source code, so we don't need to keep all possible builds over time.
MILESTONE 1
- Basic pack files (like git)
MILESTONE 2
- Store checksums as ay
- Drop version/metadata from tree/dirmeta objects
- Add index size to superindex, pack size to index
- So pull can calculate how much we need to download
- Split pack files into metadata/data
- pull: Extract all we can from each packfile one at a time, then delete it
- Restructure repository so that links can be generated as a cache; i.e. objects/raw, pack files are now the canonical
- For files, checksum combination of metadata variant + raw data
- i.e. there is only OSTREE_OBJECT_TYPE_FILE (again)
MILESTONE 3
- Drop archive/raw distinction - archive repositories always generate packfiles per commit
- Include git packv4 ideas:
- metadata packfiles have string dictionary (tree filenames and checksums)
- data packfiles match up similar objects
- Rolling checksums for partitioning large files? Kernel debuginfo
- Improved pack clustering
- file fingerprinting?
- ELF-x86 aware deltas
Related work in storage
git: http://git-scm.com/ Venti: http://plan9.bell-labs.com/magic/man2html/6/venti Elephant FS: http://www.hpl.hp.com/personal/Alistair_Veitch/papers/elephant-hotos/index.html
Compression
xdelta: http://xdelta.org/ Bsdiff: http://www.daemonology.net/bsdiff/ xz: http://tukaani.org/xz/