repo: Add APIs for devino optimization between checkout -> commit

A fast way to generate new OSTree content using an existing
tree is to checkout (as hard links), add/replace files, then
call `ostree_repo_scan_hardlinks()`, then commit.

But `ostree_repo_scan_hardlinks()` scans the entire repo, which
can be slow if you have a lot of content.

All we really need is a mapping of (device,inode) -> checksum
just for the objects we checked out, then use that mapping
for commits.

This patch adds API so that callers can create a mapping via
`ostree_repo_devino_cache_new()`, then pass it to
`ostree_repo_checkout_tree_at()` which will populate it, and then
`ostree_repo_write_directory_to_mtree()` can consume it.

I plan to use this in rpm-ostree for package layering work.

Notes:
 - The old `ostree_repo_scan_hardlinks()` API still works.
 - I tweaked the cache to be a set with the checksum colocated with
   the key, to avoid a separate malloc block per entry.

https://github.com/GNOME/ostree/pull/167
This commit is contained in:
Colin Walters committed 2016-01-07 14:19:12 -05:00
1 parent 21fbc16bc3
commit 5929ce9e0e
6 files changed
+157 -51

No files matched your search

+56
View File
@@ -429,6 +429,26 @@ checkout_one_file_at (OstreeRepo *repo,
TRUE, &did_hardlink,
cancellable, error))
goto out;
if (did_hardlink && options->devino_to_csum_cache)
{
struct stat stbuf;
OstreeDevIno *key;
if (TEMP_FAILURE_RETRY (fstatat (destination_dfd, destination_name, &stbuf, AT_SYMLINK_NOFOLLOW)) != 0)
{
glnx_set_error_from_errno (error);
goto out;
}
key = g_new (OstreeDevIno, 1);
key->dev = stbuf.st_dev;
key->ino = stbuf.st_ino;
memcpy (key->checksum, checksum, 65);
g_hash_table_add ((GHashTable*)options->devino_to_csum_cache, key);
}
if (did_hardlink)
break;
}
@@ -834,6 +854,42 @@ ostree_repo_checkout_tree_at (OstreeRepo *self,
return ret;
}
static guint
devino_hash (gconstpointer a)
{
OstreeDevIno *a_i = (gpointer)a;
return (guint) (a_i->dev + a_i->ino);
}
static int
devino_equal (gconstpointer a,
gconstpointer b)
{
OstreeDevIno *a_i = (gpointer)a;
OstreeDevIno *b_i = (gpointer)b;
return a_i->dev == b_i->dev
&& a_i->ino == b_i->ino;
}
/**
* ostree_repo_devino_cache_new:
*
* OSTree has support for pairing ostree_repo_checkout_tree_at() using
* hardlinks in combination with a later
* ostree_repo_write_directory_to_mtree() using a (normally modified)
* directory. In order for OSTree to optimally detect just the new
* files, use this function and fill in the `devino_to_csum_cache`
* member of `OstreeRepoCheckoutOptions`, then call
* ostree_repo_commit_set_devino_cache().
*
* Returns: (transfer full): Newly allocated cache
*/
OstreeRepoDevInoCache *
ostree_repo_devino_cache_new (void)
{
return (OstreeRepoDevInoCache*) g_hash_table_new_full (devino_hash, devino_equal, g_free, NULL);
}
/**
* ostree_repo_checkout_gc:
* @self: Repo