I'm aiming to do some more work on the Rust side around `fsck`
like functionality, and this is a useful primitive. There isn't
a great Rust crate for xattrs, and I think it's better to share this
code.
This adds two new object types for storing xattrs separately from
content objects.
`.file-xattrs` are regular files storing xattrs content, encoded as
GVariant. Each object is keyed by the checksum of its content, allowing
for multiple references.
`.file-xattrs-link` are hardlinks which are associated to file objects.
Each object is keyed by the same checksum of the corresponding file
object. The target of the hardlink is an existing file-xattrs object.
In case of reaching the limit of too many links, this object could be
a plain file too.
The commit metadata `version` key is well established but getting it for
a remote commit is cumbersome since the commit object needs to be
fetched and loaded. Including it in the summary additional metadata
allows a much more convenient view of what each of the remote refs
represents.
Add a standard key for this. We actually had a case in OpenShift
builds recently where a `ppc64le` image was pushed over an `x86_64`
one and this started failing at runtime with a not immediately
obvious error.
I'll probably end up changing rpm-ostree at least to use
the RPM architecture for this key and fail if it doesn't match
the booted value.
Possibly that should live in ostree but it would involve adding
architecture schema here, which gets into a big mess. Let's
just standardize the key.
xref e02ef2683d
These had been added assuming 2019.7 would be the next version, but now
it's 2020 and there's been a release. In the case of
`OstreeCommitSizesEntry`, I'd forgotten to move it forward from 2019.5
to 2019.7 in the time between when I started working on the feature and
it landed.
This function parses the object listing in the `ostree.sizes` metadata
and returns an array of `OstreeCommitSizesEntry` structures.
Unfortunately, for reasons I don't understand, the linker wants to
resolve `_ostree_read_varuint64` from `ostree-core.c` even though it's
not used by `test-checksum.c` at all.
There are use cases for libostree as a local content store
for content derived or delivered via other mechanisms (e.g. OCI
images, RPMs, etc.). rpm-ostree today imports RPMs into OSTree
branches, and puts the RPM header value as commit metadata.
Some of these can be quite large because the header includes
permissions for each file. Similarly, some OCI metadata is large.
Since there's no security issues with this, support committing
such content.
We still by default limit the size of metadata fetches, although
for good measure we make this configurable too via a new
`max-metadata-size` value.
Closes: https://github.com/ostreedev/ostree/issues/1721Closes: #1744
Approved by: jlebon
Currently the API that allows P2P operations (e.g. pulling an ostree ref
from a LAN or USB source) is hidden behind the configure flag
--enable-experimental-api. This commit makes the API public and makes
that flag essentially a no-op (leaving it in place in case we want to
use it again in the future). The P2P API has been tested over the last
several months and proven to work.
This means that since we're no longer using the "experimental" feature
flag, P2P builds of Flatpak will fail when using versions of OSTree from
this commit onwards, until Flatpak is patched in the near future. If you
want to build Flatpak < 0.11.8 with P2P enabled and link against OSTree
2018.6, you'll have to patch Flatpak. However, since Flatpak won't yet
have a hard dependency on OSTree 2018.6, it needs a new way to determine
if the P2P API in OSTree is available, so this commit adds a "p2p"
feature flag. This way the feature set is more semantically correct than
if we had continued to use the "experimental" feature flag.
In addition to making the P2P API public, this commit makes the P2P unit
tests run by default, removes the f27-experimental CI instance that's no
longer needed, changes a few man pages to reflect the changes, and
updates the bash completion script to accept the new commands and
options.
Closes: #1596
Approved by: cgwalters
It will be used by successive commits to keep track of the payload
checksum for objects stored in the repository.
The goal is that files having the same payload but different xattrs can
take advantage of reflinks where supported.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #1443
Approved by: cgwalters
There are a few cases for knowing whether a commit has identical
content to another commit. Some people want to do a "promotion workflow",
where the content of a commit on a tesitng branch is then "promoted"
to a production branch with `ostree commit --tree=ref`.
Another use case I just hit in rpm-ostree deals with
[jigdo](https://github.com/projectatomic/rpm-ostree/issues/1081) where we're
importing RPMs on both the client and server, and will be using the
content checksum, since the client/server cases inject different metadata
into the commit object.
Closes: https://github.com/ostreedev/ostree/issues/1315Closes: #1449
Approved by: jlebon
SPDX License List is a list of (common) open source
licenses that can be referred to by a “short identifier”.
It has several advantages compared to the common "license header texts"
usually found in source files.
Some of the advantages:
* It is precise; there is no ambiguity due to variations in license header
text
* It is language neutral
* It is easy to machine process
* It is concise
* It is simple and can be used without much cost in interpreted
environments like java Script, etc.
* An SPDX license identifier is immutable.
* It provides simple guidance for developers who want to make sure the
license for their code is respected
See http://spdx.org for further reading.
Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
Closes: #1439
Approved by: cgwalters
This is a freeform string useful to track/display when a commit is "derived"
from some other format. For example, in the rpm-ostree test we make a
`vmcheck` ref that conceptually overlays the default ref like
`fedora-atomic:fedora/26/x86_64/atomic-host`.
My current patch sets the source title to e.g.
"Dev overlay on fedora-atomic:fedora/26/x86_64/atomic-host".
Another case I'm working on now is importing OCI images to use
as host images. For that case, the source title is
With this patch we could then set the original OCI image name + tag
as the source name, like:
"oci:cgwalters/demo-custom-fedora-atomic-host:26".
Closes: #1296
Approved by: jlebon
This is like `ostree_checksum_file` but fd-relative. This will be used
by https://github.com/ostreedev/ostree/pull/1258.
AFAICT, we actually didn't have any tests that check the `checksum` CLI.
Add a basic one here to test the old code as well as the new code.
Closes: #1263
Approved by: cgwalters
For the old `OSTREE_REPO_MODE_ARCHIVE_Z2`. Use it mostly tree
wide except for the repo finder tests (to avoid conflicting with
some outstanding PRs).
Just noted another user coming in some of those tests and wanted to do a
cleanup.
Closes: #1209
Approved by: jlebon
We added a `.dir-locals.el` in commit: 9a77017d87
There's no need to have it per-file, with that people might think
to add other editors, which is the wrong direction.
Closes: #1206
Approved by: jlebon
These were previously private, but since we expect people to use them, let's add
`#define`s like we did for some of the other commit metadata.
Closes: #1028
Approved by: jlebon
This is a type representing the tuple (collection ID, ref name), which is
guaranteed to be globally unique. It will be used in upcoming commits.
It introduces the concept of a ‘collection’ which is a unique, curated
set of refs which lie in the same trust domain (i.e. all signed by the
same key and validated by the same developer). Flathub might be a
collection, for example; or the set of OS refs coming from a particular
OS vendor.
It includes a function for validating collection IDs.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
There are a few places in the code where ad-hoc validation was being
performed. Might as well formalise it a bit more.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #948
Approved by: cgwalters
This was making it impossible to pull or mirror a large ostree repo, and
according to Colin is no longer necessary. It works fine with a test
against a repo with 2741 commit and 451468 objects in it.
Closes: #899Closes: #904
Approved by: jlebon
• Commit timestamps, so it’s easy to work out whether a given commit is
newer than the one we have locally
• Summary file timestamp, so it’s easy to work out whether the summary
file is more up to date than another summary file
• Summary file expiry time, so clients can work out when they should
expect the summary file to next be updated, and hence can query for
it at roughly the right time
The expiry time requires input from the user, so is currently never set
automatically. Programs using libostree can set it if they wish.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #826
Approved by: cgwalters
This mode is similar to bare-user, but does not store the permission,
ownership (uid/gid) and xattrs in an xattr on the file objects in the
repo. Additionally it stores symlinks as symlinks rather than as
regular files+xattrs, like the bare mode. The later is needed because
we can't store the is-symlink in the xattr.
This means that some metadata is lost, such as the uid. When reading a
repo like this we always report uid, gid as 0, and no xattrs, so
unless this is true in the commit the resulting repository will
not fsck correctly.
However, it the main usecase of the repository is to check out with
--user-mode, then no information is lost, and the repository can
work on filesystems without xattrs (such as tmpfs).
Closes: #750
Approved by: cgwalters
[Previously](https://github.com/ostreedev/ostree/pull/728) we added compile-time
checking for versions, but there are use cases for runtime checking as well,
because in a number of API calls we use `GVariant` as an API extension
mechanism.
Closes: #735
Approved by: jlebon
Add a ostree_raw_file_to_archive_z2_stream_with_options() variant of
ostree_raw_file_to_archive_z2_stream(), to allow a compression-level
option to be passed in and passed through to zlib.
This is useful when building archive-z2 files on the fly for
transmission over a non-bandwidth-limited channel, such as a local
network. In this case, CPU time is more valuable than bandwidth, so we
want a low compression level.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #721
Approved by: cgwalters
This way e.g. flatpak can detect which timestamp it should use.
See `flatpak/common/flatpak-utils.c:flatpak_zero_mtime`.
Closes: #501
Approved by: cgwalters
The checksum_b64_inplace variants can't be used in bindings. Provide
versions that allocate and return the output rather than working on a
passed in buffer. These can then be used in GI bindings to get the
ostree modified base64 encodings.
Closes: #398
Approved by: cgwalters
This changes around a few things that didn't work for me:
* Section names seem to be ostree-* instead of libostree-*
* Also XML files are ostree-* (they didn't show up at all)
- gtk-doc doesn't seem to parse const _OSTREE_PUBLIC correctly
* pull documentation is now on the actual functions rather than stubs
* Update gitignore with some more files
And there some changes to make gtk-doc give fewer warnings (not finished)
Closes: #327
Approved by: cgwalters
It is quite similar to the already existing
ostree_raw_file_to_content_stream function, so I factored the common
part to a separate function. The difference is that we cannot report
the size of the resulting stream.
Can be useful for serving a "bare" repository as a faked "archive-z2"
repository.
Closes: #308
Approved by: cgwalters
Use markdown to actually format the tuple member descriptions as a
list. Currently they are hardly legible as gtk-doc formats them into a
single sentence.
Also update the additional metadata docs in the summary objects.
Closes: #289
Approved by: cgwalters
If a symbol falls in a git merkle tree forest, but no one hears it,
did it ever exist?
I'm claiming this isn't an API/ABI break because nothing could
have actually used this. `git log -S checksum_update_meta` leads
me to 38ef75e6e0 which was before
we really had a stable shared library.
Closes: #263
Approved by: giuseppe
As rpm-ostree evolves, it keeps driving API additions to libostree.
This creates a relatively tight coupling.
However, if delivering via e.g. RPM, unless one manually remembers to
increment the `Requires:` in the spec file, it's possible for the two
to become desynchronized.
RPM handles versioned symbols and will ensure a dependency if the
application starts using a newer version.
To implement this, switch to `-fvisibility=hidden`, along with an
annotation in the header, and finally add a `.sym` file.
This matches what other projects like systemd and libvirt do.
Although rather than attempting to retroactively version symbols, glom
them all onto the current one.
Add a new object type: OSTREE_OBJECT_TYPE_TOMBSTONE_COMMIT that is
used when a commit was intentionally removed.
If the remote repository doesn't use tombstone commits, do not fail on
a missing commit (change 0b795785dd).
When the remote repository uses tombstones, if a commit cannot be
found, check if the tombstone file is present and fail if it is not
present.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
This will be used for a later change to use openat() for the fetching
code. Note that we drop the code to use mmap() - it was an attempt to
avoid keeping a fd open, but we do correctly close anyways.
This format is pretty much the same as the "bare" format, except the
file ownership and xattrs is not stored in the actual filesystem object, but
rather on the side in a user xattr. This means two things:
1) An unprivileged user can store such a repo independent of the types
of files in it or their xattrs. And you can later (as root)
reconstruct the real filesystem tree with ownership. Although you
can't do that using hardlink-sharing. This also means ostree
fsck does a full verification.
2) Such a repository can be checked out with user-mode (checkout -U)
as an unprivileged user using hardlinks for space sharing.
Additionally, symlinks are stored as regular files (with the content
being the symlink target) because user xattrs are not supported on
symlinks. We know at checkout time if the file is a symlink because
the original st_mode is stored in the xattr metadata.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
For Fedora and potentially other distributions which use globally
distributed mirrors, metalink is a popular solution to redirect
clients to a dynamic set of mirrors.
In order to make metalink work though, it needs *one* file which can
be checksummed. (Well, potentially we could explode all refs into the
metalink.xml, but that would be a lot more invasive, and a bit weird
as we'd end up checksumming the checksum file).
This commit adds a new command:
$ ostree summary -u
To regenerate the summary file. Can only be run by one process at a
time.
After that's done, the metalink can be generated based on it, and the
client fetch code will parse and load it.
https://bugzilla.gnome.org/show_bug.cgi?id=729585
If fetching GPG-signed commits over plain HTTP, a MitM attacker can
fill up the drive of targets by simply returning an enormous stream
for the commit object.
Related to this, an attacker can also cause OSTree to perform large
memory allocations by returning enormous GVariants in the metadata.
This helps close that attack by limiting all metadata objects to 10
MiB, so the initial fetch will be truncated.
But now the attack is only slightly more difficult as the attacker
will have to return a correctly formed commit object, then return a
large stream of < 10 MiB dirmeta/dirtree objects.
https://bugzilla.gnome.org/show_bug.cgi?id=725921
We don't want to allow MITM attackers to intercept upgrade requests
and provide clients with older OS versions vulnerable to security
flaws.
Only "ostree admin upgrade" gets this behavior for now - whether we
want to do it for "ostree admin switch" is another question.
Colin Walters
Luca BRUNO
Dan Nicholson
Joseph Marrero
Dan Nicholson
Matthew Leeds
Giuseppe Scrivano
Marcus Folkesson
Jonathan Lebon
Ruixin
Philip Withnall
Brian C. Lane
Alexander Larsson
Mathnerd314
Krzesimir Nowak