Commit Graph

374 Commits

Author SHA1 Message Date
Colin Walters addbdb619c ostbuild: Stop generating git mirror config
Nothing's using it right now and it's a little buggy.
2012-03-13 10:39:31 -04:00
Colin Walters 87e9a29f11 ostbuild: Further work on regenerating a snapshot from a compose 2012-03-13 10:39:26 -04:00
Colin Walters 07e93bd326 ostbuild: Change to name-prefix and base-prefix in manifest
This makes it clearer that we're using this as a prefix, when we
generate the snapshots.
2012-03-13 10:39:26 -04:00
Colin Walters 76f7bf2060 ostbuild: Switch to composing via checkouts+commit
Store the compose contents as a JSON file in the filesystem tree.
This is more flexible compared to defining "compose" as just a literal
layering of filesystem trees.  For example, we might want to run
prelink.
2012-03-13 10:39:26 -04:00
Colin Walters 5dd62af628 ostbuild: Stop using "compose" to make buildroots
It pollutes the repository with a combinatorial explosion of .dirtree
metadata files, and we don't really need to track buildroots at that
level.
2012-03-13 10:39:25 -04:00
Colin Walters 5275e8d6b2 ostbuild: Drop "commit-artifacts" builtin
It's long since dead.
2012-03-13 10:39:25 -04:00
Colin Walters c11cefb99b ostbuild: Kill "autodiscover-meta"
No longer needed.
2012-03-13 10:39:25 -04:00
Colin Walters d3059d254b ostbuild: Adapt "status" to expanded manifest 2012-03-13 10:39:25 -04:00
Colin Walters ff81b21bcf ostbuild: Adapt "build" to newly updated expanded manifest 2012-03-13 10:39:25 -04:00
Colin Walters c71770c08c ostbuild: Write out separate arch-component snapshots 2012-03-13 10:39:25 -04:00
Colin Walters c3debe0fb6 ostbuild: 'checkout' command now also applies patches 2012-03-13 10:39:25 -04:00
Colin Walters d8eebaa2d1 ostbuild: Pull patches from specified patches git repository
We need to be more formal about where we get patches from, and more
specifically what version.  Let's assume they're also stored in a git
repository that we fetch, instead of copying them from "wherever the
manifest is".

This meshes nicely with splitting between manifest.json and
snapshot.json.
2012-03-13 10:39:25 -04:00
Colin Walters 6890444f28 ostbuild: Change metadata keys for artifacts
"ostbuild-name" is important to tie back in to the snapshot.
"ostbuild-revision" is shorter than ostbuild-artifact-version.
2012-03-13 10:39:25 -04:00
Colin Walters 12f2f8929d ostbuild: Copy snapshot JSON into artifact trees
This is kind of a hack...but...
2012-03-13 10:39:25 -04:00
Colin Walters d2f298488f ostbuild: Sort keys in JSON snapshot 2012-03-13 10:39:25 -04:00
Colin Walters b8e3f622f1 ostbuild: Rename build manifest to "snapshot.json"
It was confusing having two manifests.  "snapshot" matches the
jhbuild term for the same thing.
2012-03-13 10:39:24 -04:00
Colin Walters 40aedeb7a4 ostbuild: Move more intelligence into chroot-compile-one
Let's have chroot-compile-one be the core logic for composing a
buildroot, etc.  This allows us to use it as a developer tool by
checking out a tree, then doing a chroot build.
2012-03-13 10:39:24 -04:00
Colin Walters db93c43255 core: Fix "cat" crashing when it should have returned ENOENT 2012-03-08 12:10:58 -05:00
Colin Walters eb4302517f core: Fix crasher in "rev-parse" 2012-03-08 12:10:58 -05:00
Colin Walters 76bc35186e core: Add --union mode to checkout
This is another step towards ostbuild using this instead of the
"compose" builtin.
2012-03-06 11:59:06 -05:00
Colin Walters 83fb6d56e1 core: Don't call unlink() on directory if it already exists
If we fail to make a directory because it exists, we shouldn't call
unlink() on it.
2012-03-06 11:59:06 -05:00
Colin Walters 1513f29495 core: Pull diff functionality out into "diff" builtin
There's no good reason for this to be in core when it's only in use by
the diff builtin.
2012-03-06 11:59:06 -05:00
Colin Walters 4db485dd5f core: Add "subpath" option to checkout
Will be used by ostbuild to extract e.g. /runtime or /devel from
artifacts.
2012-03-06 11:59:06 -05:00
Colin Walters 1f7d776a18 core: Improve checkout API
Expose the lower-level functionality in libostree, change checkout
builtin to be a higher level driver.  This will allow us to more
easily improve the "checkout" builtin..
2012-03-06 11:59:06 -05:00
Colin Walters f44b8aca44 core: Add ostree-types.h
This allows us to have circular references between the headers.
2012-03-06 11:59:06 -05:00
Colin Walters f484665bfe libotuil: Fix missing #define in header 2012-03-06 11:59:06 -05:00
Colin Walters e4849ebd2b mainloop: Don't create poll until we need to
This slightly optimizes the case where we speculatively create a
Mainloop we might not use.
2012-03-06 11:59:06 -05:00
Colin Walters da00891688 core: Allow 'rev-parse' to resolve multiple revs 2012-03-06 11:59:06 -05:00
Colin Walters eb7c3d01a3 core: Add 'cat' builtin 2012-03-06 11:59:06 -05:00
Colin Walters 9dc86d0f5e compose: Don't print trailing (null) if we skip committing
If we skip a commit due to an idential content parent, we shouldn't
print the (null) commit checksum.
2012-03-01 16:08:27 -05:00
Colin Walters 6829ab2260 pull: Allow pulling multiple specific refs 2012-02-28 13:17:03 -05:00
Colin Walters 19bc7ff85f prune: Fix two crasher bugs 2012-02-27 09:53:44 -05:00
Colin Walters 2440338968 pull: Add ability to fetch refs/summary if no ref specified
This allows us to fetch all refs, which is useful for ostbuild where
we do want the buildroot/ refs.
2012-02-27 08:37:51 -05:00
Colin Walters b783ebc189 core: Add API to validate rev 2012-02-27 08:37:37 -05:00
Colin Walters 810a3dac86 core: Write a refs/summary file for archive repositories
This will allow us to retrieve all refs over HTTP easily.
2012-02-27 06:20:42 -05:00
Colin Walters d3a73225b0 core: Add API to enumerate all refs 2012-02-27 05:58:42 -05:00
Colin Walters 6ff07e0b3e pull: Print download URLs by default, rather than requiring -v
Let's leave verbose for more debug-type stuff.
2012-02-26 08:35:26 -05:00
Colin Walters 5329f1706b core: Further fix fallout from archive mode files
Continuing from 16c0cfe9b5, we now have
the problem that we're removing the executable bit, which obviously
breaks things when we chroot in for a build.  Fix this by masking over
our bits.
2012-02-24 17:50:44 -05:00
Colin Walters 0e5919bde1 core: Disable Soup-gnome features for pull
Otherwise we get a dconf thread that tries to talk to the session bus.
I think this comes from the proxy settings lookup.
2012-02-24 17:26:17 -05:00
Colin Walters ab58faefef core: Disable Gvfs for binaries
Otherwise
1) Lots of unnecessary shit happens and I like my straces to be clean
2) There is no dbus session bus for system daemons or when we're run
   in a root context
2012-02-24 17:24:50 -05:00
Colin Walters 51916a7002 core: Look up refs like origin/foo correctly again
We want to support both "bare" lookups where "foo" can be local, or in
any remote, as well as prefixed ones for a specific remote.

This fixes ostree-pull noticing that nothing has changed.
2012-02-24 16:08:34 -05:00
Colin Walters a417ee3fed core: Add "prune" builtin
This should be useful on clients to trim old refs.  For example,
after an upgrade the system could do:

ostree --repo=/ostree/repo prune --depth=2 gnomeos-3.4-i686-runtime

This would remote all objects that aren't in the current build and the
previous one.
2012-02-24 10:23:35 -05:00
Colin Walters 16c0cfe9b5 core: Fix regression when committing symlinks in archive mode
Commit 802959f398 broke this because it
made all archived content files be S_IFREG, which is clearly wrong.
2012-02-24 10:05:26 -05:00
Colin Walters 14cda593d5 core: Correctly look up refs in remotes 2012-02-23 23:52:59 -05:00
Colin Walters 8145faecbf pull: Don't automatically uncompress data
This totally breaks when the file we're trying to request was already
gzip'd.  We don't want to uncompress e.g. man pages because that
breaks content-addressed storage.
2012-02-23 21:52:15 -05:00
Colin Walters 7907a853b4 pull: Incrementally write into temporary file
Rather than downloading into one big chunk in memory, then writing all
at once.
2012-02-23 19:20:36 -05:00
Colin Walters b2ad353b42 core: Store "pending" objects explicitly, fix pull to use them
Before we were creating randomly-named temporary files in repo/tmp
when downloading via pull, but that means if the download process is
interrupted, we have to redownload everything again.

Let's still keep the concept of a "transaction" where files are
stored in the repository as atomically as possible (i.e. we
do a bunch of rename() calls), but now we also have an explicit
"tmp/pending/objects" directory that contains named objects.

This allows us to then skip redownloading things that are pending.
2012-02-23 18:19:00 -05:00
Colin Walters 802959f398 core: Ensure .archive-content files are mode 0644
We don't want to inherit the permissions of the actual file, otherwise
trying to fetch e.g. /etc/passwd when being served via Apache httpd
will break.
2012-02-23 12:34:43 -05:00
Colin Walters dee30bda7c pull: Only fetch .archive-content for regular files
Otherwise e.g. Apache httpd might try to follow a symbolic link to
/bin/tinylogin which clearly won't work.
2012-02-23 12:30:01 -05:00
Colin Walters 0a195e6874 ostbuild: Make resolve fully expand config-opts, delete now-unnecessary vcsconfig
The source manifest format allows for compression, but it's more
convenient for consuming code if it's fully expanded in one place.
2012-02-22 15:17:42 -05:00
Colin Walters b26b981b0d ostbuild: Use correct directory for submodule update 2012-02-22 14:17:18 -05:00
Colin Walters 9939f1861d osbuild: Don't use linux-user-chroot for git submodules
On the ostree.gnome.org build server, LDAP is used for username
lookups, and git aborts if it can't look up the user.
2012-02-22 11:51:18 -05:00
Colin Walters 341096f18a ostbuild: Drop down to Python 2.6
Hopefully this gets us working on RHEL6.
2012-02-20 22:31:10 -05:00
Colin Walters 21f1ca364f triggers: Fix comparison function 2012-02-19 21:10:59 -05:00
Colin Walters 633da1e7ad triggers: We need to order their execution
Otherwise we run in inode order which is unpredictable.  In particular
this causes problems because we might run e.g. the gtk+ trigger before
the gdk-pixbuf one.  And ldconfig should really be first.
2012-02-19 20:52:50 -05:00
Colin Walters f1e38d6e4b ostbuild: Add --recompose argument
This is necessary to run after rebuilding the Yocto base.
2012-02-16 18:30:42 -05:00
Colin Walters 8729997eab switchroot: Bind mount /lib/modules after we mount /lib
Otherwise we don't ge the right /lib/modules.
2012-02-13 10:56:13 -05:00
Colin Walters fcd70e5891 switchroot: Bind mount /lib/modules from /ostree/modules
Since we have kernels outside of the tree, keep modules outside
too.
2012-02-10 19:17:29 -05:00
Colin Walters 351d243ccf ostbuild: Fix --debug-shell
We want to reuse an existing source tree.
2012-02-09 16:47:25 -05:00
Colin Walters 820709e791 ostbuild: When doing a fetch, call fetch before trying to mirror the branch
Otherwise when the manifest specifies a branch we haven't fetched, we
error out.
2012-02-09 11:19:33 -05:00
Colin Walters feca75a930 switchroot: remount rootfs read/write
See rationale in comment.
2012-02-08 22:23:25 -05:00
Colin Walters 8e944cbffd ostree-switch-root: Correctly prepend sysroot before mounts to move from dracut 2012-02-08 20:22:27 -05:00
Colin Walters 0172ab5a3a switchroot: Further work on being dracut-only 2012-02-08 17:32:34 -05:00
Colin Walters fef67756f9 ostbuild: Add 'status' command
Roughly equivalent to 'jhbuild info'.
2012-02-08 16:37:14 -05:00
Colin Walters a904c9693b Add ostree-switch-root here
Merge the code from ostree-init; now that we're back to targeting an
initramfs (dracut), we don't need to statically link the binary, so
there's no strong reason to have a separate module.
2012-02-05 16:02:44 -05:00
Colin Walters 097c1b8025 vcs: Delete temporary directory too 2012-02-02 13:56:46 -05:00
Colin Walters 73fa98694e builtins: Fix error message if workdir doesn't exist 2012-02-02 13:56:25 -05:00
Colin Walters 845ba39a0e ostbuild: Add 'checkout' builtin
This is useful for the developer to get the source code of
several things at exactly the revision specified in the manifest.
2012-02-02 09:22:44 -05:00
Colin Walters df92b8d46f ostbuild: Unify mirrordir lookup code 2012-02-02 08:40:18 -05:00
Colin Walters f1930d2c99 triggers: Fix typo in gconf trigger 2012-01-31 22:59:28 -05:00
Colin Walters a8151ac836 Add gconf trigger 2012-01-31 22:54:23 -05:00
Colin Walters 914f9ffc61 ostbuild: Actually compose devel image from devel base
Otherwise we don't have gcc, gdb, git etc...
2012-01-31 18:57:02 -05:00
Colin Walters 102f6a7f43 ostbuild: Make separate .lastfetch files for e.g. gtk2 and gtk3
This ensures we don't need to check out any git repository which is
built twice under different names.
2012-01-29 17:24:20 -05:00
Colin Walters e048173c60 ostbuild: Fix error message for unknown builtin 2012-01-29 16:47:19 -05:00
Colin Walters 8c878ebb86 mainloop: Ensure we reset running to True when rerun
Otherwise we pretty much exit immediately...
2012-01-29 13:29:31 -05:00
Colin Walters 3426de05fe ostbuild: Ensure we return the right process exit code 2012-01-27 11:09:53 -05:00
Colin Walters 1d3b9ae711 triggers: Add dconf update
Necessary for gdm at least.
2012-01-23 18:09:33 -05:00
Colin Walters 00ec0ddb9b core: Add --delete option for fsck
Useful to clean up any broken objects.
2012-01-23 17:06:10 -05:00
Colin Walters 8e36e80896 ostbuild: Support marking files as setuid 2012-01-22 13:31:22 -05:00
Colin Walters e8dbaa3c07 core: Add generic "commit filter" API, use it to implement statoverride
The builder wants the ability to mark a given file as e.g. setuid.  To
implement this, the repo now has a callback-based API when importing a
directory to modify or remove items.

The commit tool accepts a "statoverride" file as input which looks like:

+mode /path/to/file
2012-01-22 13:27:38 -05:00
Colin Walters 1ca517f36f ostbuild: chdir into source directory when launching debug shell 2012-01-20 17:52:08 -05:00
Colin Walters dde4da9513 core: Allow cloning only certain refs for local clone
This is way, way faster.
2012-01-20 17:50:59 -05:00
Colin Walters d4470a844d ostbuild: Ensure we look in correct git repository if --fetch is specified
The mirrordir variable was bound from the previous iteration, which
was clearly broken.
2012-01-19 18:31:37 -05:00
Colin Walters 16f2f5f7e5 mainloop: Only delete pid watch if we found the right pid 2012-01-19 18:30:45 -05:00
Colin Walters abfe17e2cb ostbuild: Show cwd on failure
This is helpful for disambiguation.
2012-01-19 16:41:59 -05:00
Colin Walters 39fe050f04 core: Add standard::size to info we get from archives
This makes "ostree ls" show the file size.
2012-01-19 09:54:09 -05:00
Colin Walters 660ff7422f ostbuild: Various cleanups 2012-01-18 17:38:31 -05:00
Colin Walters 970682a779 ostbuild: Actually do check for Makefile
Just kidding, we need the makefile path to check for .NOTPARALLEL
2012-01-18 17:08:30 -05:00
Colin Walters 4af64c7635 ostbuild: Don't check for a "Makefile"
WebKit uses GNUmakefile which is totally valid, and this check
isn't really gaining us anything.
2012-01-18 16:40:47 -05:00
Colin Walters 55f6998d37 ostbuild: Ensure we don't retrieve submodules from the network
This is a bit gross...we have to manually frob the submodule
URLs to point to our mirror.
2012-01-18 13:35:53 -05:00
Colin Walters 298e4957d9 ostbuild: Support rm-configure flag
This is easier than making git patches that delete configure.
2012-01-16 20:17:39 -05:00
Colin Walters 9d37f76a5a ostbuild: Actually exit with error if we fail a build 2012-01-16 17:50:57 -05:00
Colin Walters 5bf4eb39e7 ostbuild: Do fetches before resolution
Otherwise we get the OLD revision in the manifest.
2012-01-16 09:57:40 -05:00
Colin Walters a6e6146ab6 core: Note which artifact failed to compose 2012-01-16 09:57:24 -05:00
Colin Walters 813923943d ostbuild: Fix documentation dir (should be 'doc') 2012-01-16 09:57:00 -05:00
Colin Walters b1ff693ea3 Revert "ostbuild: Check for the Makefile in the correct directory"
I'm not sure how this made anything work...probably I was screwing
something else up.

This reverts commit aeeed8da13.
2012-01-16 09:39:13 -05:00
Colin Walters bff6baced8 ostbuild: Save build logs to a file, monitor output for warnings
This is bringing us back closer to metabuild.
2012-01-16 09:27:46 -05:00
Colin Walters d5def94b54 ostbuild: Clean up artifact splitting, put non-symlink .so in runtime
We were breaking nss which just installs libnss3.so as the real
shared library.
2012-01-15 21:45:12 -05:00
Colin Walters e34b90453c gnomeos: Building gdm 2012-01-15 17:06:18 -05:00
Colin Walters ef7f12f31b ostbuild: Run triggers after constructing build root 2012-01-15 17:05:23 -05:00