Prep for reworking how we do sysroot cleanup. We're going to
start doing more lowlevel pruning work there, and I wanted to avoid
duplicating the ref enumeration.
Closes: #1566
Approved by: jlebon
This is easier to `git grep` etc. versus ad-hoc English. Although
we still have some English for the prepare_transaction/commit which
acquire/release in separate phases.
Closes: #1572
Approved by: jlebon
The code has been sitting around for a while but since I disabled
it by default, I doubt anyone is really using it or relying on it.
This patch and turns on locking by default, and also drops the
API which was only public in the experimental API builds.
Conceptually these are two distinct things, and we
may actually want to split up the patches.
I don't think this will break anyone, but it's hard to say for sure.
It's also going to be hard to find out until we actually release
I suspect...
But anyone who is broken should be able to add `locking=false` into
their repo config. On the flip side Endless has been shipping with
this enabled and it is reported to help.
The reason to drop the APIs: I'm a bit concerned about the interactions over time
between libostree's use of the API and any apps that start using it.
For example, if an app specifies a SHARED lock in their code, then
later internally we decide to temporarily grab an `EXCLUSIVE`, but the
app had a second thread/process that was `EXCLUSIVE` already, and
that process was waiting on the first bit of code, then we could
deadlock. I can't think of a real world situation where this would happen
yet though.
We are likely to in the future have say `fsck` take an external lock,
`checkout` grab a shared one, etc.
Closes: #1555
Approved by: jlebon
When a new object is added to the repository, create a
$PAYLOAD-SHA256.payload-link symlink file as well. The target of the
symlink is the checksum of the object that was added the repository.
Whenever we add a new object file, in addition to lookup if the file is
already present with the same checksum we also check if an object with
the same payload is in the repository.
If a file with the same payload is already present in the repository, we
copy it with `glnx_regfile_copy_bytes` that internally attempts to
create a reflink (ioctl (..., FICLONE, ..)) to the target file if the
file system supports it. This enables to have objects that share the
payload but have a different inode and xattrs.
By default the payload-link-threshold value is G_MAXUINT64 that disables
the feature.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #1443
Approved by: cgwalters
I think this got changed in a refactor. We definitely want
to total up the amount of space that *would* be freed even
with `--no-prune` AKA `OSTREE_REPO_PRUNE_FLAGS_NO_PRUNE`.
It's actually a bit terrifying this is apparently the first test case for
the `--no-prune` option...
Closes: https://github.com/ostreedev/ostree/issues/1480Closes: #1483
Approved by: jlebon
SPDX License List is a list of (common) open source
licenses that can be referred to by a “short identifier”.
It has several advantages compared to the common "license header texts"
usually found in source files.
Some of the advantages:
* It is precise; there is no ambiguity due to variations in license header
text
* It is language neutral
* It is easy to machine process
* It is concise
* It is simple and can be used without much cost in interpreted
environments like java Script, etc.
* An SPDX license identifier is immutable.
* It provides simple guidance for developers who want to make sure the
license for their code is respected
See http://spdx.org for further reading.
Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
Closes: #1439
Approved by: cgwalters
For the [rpm-ostree jigdo ♲📦](https://github.com/projectatomic/rpm-ostree/issues/1081) work.
We're basically doing "pull" via a non-libostree mechanism, and this
should be fully supported. As I mentioned earlier we should try to
have `ostree-repo-pull.c` only use public APIs; this gets us closer
to that.
Closes: #1376
Approved by: jlebon
Add exclusive repository locking to all the pruning entry points. This
ensures that objects and deltas will not be removed while another
process is writing to the repository.
Closes: #1343
Approved by: cgwalters
We added a `.dir-locals.el` in commit: 9a77017d87
There's no need to have it per-file, with that people might think
to add other editors, which is the wrong direction.
Closes: #1206
Approved by: jlebon
We have `ot_ensure_unlinked_at()` for the "ignore ENOENT" case, and
`glnx_unlinkat()` otherwise. Port all in-tree callers to one or the other as
appropriate.
Just noticed an unprefixed error in the refs case and decided to do a tree-wide
check.
Closes: #1142
Approved by: jlebon
When working with collections it can be useful to see remote refs rather
than just local and mirrored ones. This commit changes the "ostree refs
-c" output to include remote refs, and includes remote refs with
collection IDs in summary file generation as well. The former behavior
is consistent with how "ostree refs" works, and the latter behavior is
useful in facilitating P2P updates even when mirrors haven't been
configured.
To accomplish this, OstreeRepoListRefsExtFlags was extended with an
EXCLUDE_REMOTES flag. This was done rather than an INCLUDE_REMOTES flag
so that existing calls to ostree_repo_list_refs_ext continue to have the
same behavior. This flag was added to ostree_repo_list_collection_refs
(which is an experimental API break).
Also, add unit tests for the "refs -c" and summary file behavior, and
update relevant tests.
Closes: #1069
Approved by: cgwalters
Use the new macros introduced recently in libglnx to make iterating over
hash tables cleaner. This is just a start, it does not migrate the whole
tree.
Update submodule: libglnx
Closes: #971
Approved by: cgwalters
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
Follow up to a previous patch that addressed a double-close; I
realized we already had a helper for doing "open dfd iter, do nothing
if we get ENOENT". Raise it to libotuil, and port all consumers.
Closes: #863
Approved by: jlebon
Only non-mechanical bit here was creating a local autoptr for a bit
where we'd previously done an unref for a struct member.
Closes: #847
Approved by: jlebon
There are use cases for having a single repo with branches
with different lifecycles; a simple example of what I was
trying to do in CentOS Atomic Host work is have "stable"
and "devel" branches, were we want to prune devel, but
retain *all* of stable.
This patch is split into two parts - first we add a low level "delete all
objects not in this set" API, and change the current prune API
to use this.
Next, we move more logic into the "ostree prune" command. This paves the way for
demonstrating how more sophisticated algorithms/logic could be developed outside
of the ostree core.
Also, the --keep-younger-than logic already lived in the commandline, so it
makes sense to keep extending it there.
Closes: https://github.com/ostreedev/ostree/issues/604Closes: #646
Approved by: jlebon
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.
In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.
Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.
This fixes https://github.com/ostreedev/ostree/issues/541Closes: #542
Approved by: cgwalters
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail. There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.
Closes: https://github.com/ostreedev/ostree/issues/467Closes: #471
Approved by: jlebon
When prune fails, it can be really difficult to figure out why. This at
least lets you know which objects are being considered.
https://bugzilla.gnome.org/show_bug.cgi?id=764006Closes: #224
Approved by: cgwalters
If a commit only pull has been done, then the commit object exists in
the object store in addition to the commitpartial file. Traversing this
partial commit will likely fail, but that's expected. If traverse
returns a G_IO_ERROR_NOT_FOUND in this case, continue with pruning.
https://bugzilla.gnome.org/show_bug.cgi?id=764091
In some cases (such as `ostree-sysroot-cleanup.c`), the surrounding
code would be substantially cleaner if it was also ported to
fd-relative, but I'm going to do that in a separate patch.
That way these patches are easier to review for mechanical
correctness. I used an Emacs keyboard macro as the poor man's
[Coccinelle](http://coccinelle.lip6.fr/).
Extract existing code from ostree_repo_prune and add an argument COMMIT,
that controls which commit purge. If not set, the old behavior is kept.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
First, this is just a general continuation of the `GFile -> openat`
transition.
Second, it's preparatory work for fsck to gain awareness of partial
commits.
Changes the pull API to allow pulling only a single directory instead
of the whole deployment. This option is utilized by the check-diff
option in rpm-ostree.
Add a new state directory to hold <checksum>.commitpartial files, so
we know that we've only downloaded partial state.
The prune API duplicated logic to delete objects, and furthermore the
core API to delete an object didn't clean up detached metadata.
Fix the duplication by doing the obvious thing: prune should call
_delete.
https://bugzilla.gnome.org/show_bug.cgi?id=733452
Previously I thought we'd have to ditch the current commit
format to avoid a{sv} due to
See https://bugzilla.gnome.org/show_bug.cgi?id=673012
But I realized that we don't really have to care about
unpacking/repacking commit objects, so let's just re-expose the
existing metadata a{sv} in commits in the API.
Also, add support for "detached" metadata that can be updated at any
time post-commit. This is specifically designed for GPG signatures.
https://bugzilla.gnome.org/show_bug.cgi?id=707379
Colin Walters
Giuseppe Scrivano
Marcus Folkesson
Dan Nicholson
Matthew Leeds
Jonathan Lebon
Philip Withnall
Alexander Larsson
Matthew Barnes
Anne LoVerso
Tobias Hunger