Commit Graph

348 Commits (2f533f3a68fec24372db5c29f814970c09514b67)

Author SHA1 Message Date
André Klitzing 567802749c Fix some issues with musl
* Use realpath instead of canonicalize_file_name
* Add missing include
2017-03-15 10:14:39 +01:00
Alexander Larsson 72e36b90a0 builder: Keep cache per arch
This just adds the arch name to the cache branch names, which
means a build on a different arch will not invalidate the other
arch:s build.

Fixes https://github.com/flatpak/flatpak/issues/628
2017-03-15 09:43:50 +01:00
Alexander Larsson 18f27f701b flatpak-builder --run: Always enable the dbus proxy
This means we're able to own our own id as a name, etc.
2017-03-14 16:27:51 +01:00
Alexander Larsson b6d06ba8ef Bump manifest cache version to rebuild everything
This is due to the append changes, etc
Also, we drop all _compat_ cache checksums
2017-03-14 12:02:35 +01:00
Alexander Larsson 81c444892a builder: Make cflags, cxxflags and ldflags append, not replace
This is much more natural, and means you don't have to duplicate the
flags in each place.
2017-03-14 10:53:38 +01:00
Alexander Larsson 4e2b103167 builder: Fix ordering of arch-specific build options
The arch-specific options override the generic ones,
not the other way around.
2017-03-14 10:37:12 +01:00
Alexander Larsson 0f0188aacb builder: Add support for ldflags 2017-03-14 10:26:22 +01:00
Alexander Larsson 900d146810 builder: Don't strip already stripped binaries 2017-03-09 17:03:59 +01:00
Alexander Larsson ea6747fcb4 builder: Fix --from-git
We need to specify the full branch when checking out
2017-03-07 14:20:44 +01:00
Alexander Larsson 60cadead5c builder: Properly handle relative module include paths from git
We load the sub-modules from the base directory (i.e. where
the manifest is stored).
2017-03-06 09:20:34 +01:00
Alexander Larsson 44399bcf5f builder: Add only-arches and skip-arches for sources 2017-02-28 17:41:45 +01:00
Alexander Larsson 0379c01981 builder: Add build-arches and skip-arches to modules
This lets you avoid modules on some arches.
2017-02-28 12:07:53 +01:00
Alexander Larsson c3322fd9e9 builder: Add no-make-install to modules
This is useful for modules that don't have a make install rule.
You can use the new build-commands which is run after make to
create your own custom installation phase.

Fixes https://github.com/flatpak/flatpak/issues/458
2017-02-28 11:30:59 +01:00
Alexander Larsson 352eb84189 builder: Add builder_cache_checksum_compat_boolean 2017-02-28 11:29:39 +01:00
Alexander Larsson 0b448c5f68 builder: Handle chdir failures when spawning shell 2017-02-28 11:11:28 +01:00
Alexander Larsson e9429d6759 builder: Silence unused return value warning 2017-02-28 11:05:01 +01:00
Alexander Larsson 83600f7c1b builder: Always clone git repos with transport.fsckObjects
This way we detect any corrupt (or malicious) objects
2017-02-27 16:17:57 +01:00
Alexander Larsson aaf923bd37 builder: Use separate rofiles-fuse mounts for each module
We're doing a post-commit checkout directly to the app dir, which
means the fuse filesystem cache may get out of sync with the backing
directory. So, to ensure this doesn't happen we mount a fresh rofiles
fs for each build.
2017-02-24 14:16:12 +01:00
Alexander Larsson f8388e7fdf Don't print stderr/stdout from emergency fuse unmount 2017-02-24 14:16:12 +01:00
Alexander Larsson c0da3e49b9 builder: Use splice in rofiles-fuse mount 2017-02-24 14:16:12 +01:00
Alexander Larsson 8983834f92 builder: Fix up .py[co] invalidation
Since we now only look at changed files we need to handle
the case where a .py file changed and the .pyc didn't, and
remove the now stale .pyc files.

Previously we looked at all files, so we detected this via
the stale .pyc file.
2017-02-24 14:16:12 +01:00
Alexander Larsson 364499939d builder: Add --skip-if-unchanged
This lets you skip rebuilds unless the actual json changes.
This is useful for continuos builds that only run if the json
changes, not on any commit to any git source.
2017-02-22 15:12:59 +01:00
Alexander Larsson 90674fafa0 builder: Don't mirror submodules if told not to 2017-02-22 15:12:59 +01:00
Alexander Larsson 61db31bb00 builder: Add ensure-writable to module 2017-02-22 09:48:00 +01:00
Alexander Larsson 2c71298d53 builder: Its break_hardlink, not unbreak_hardlink 2017-02-22 09:43:57 +01:00
Alexander Larsson 67dc4a153f builder: Add builder_cache_checksum_compat_strv
This lets us add strv:s without invalidating old caches
if the empty strv means same as before
2017-02-22 09:42:13 +01:00
Alexander Larsson 359d4fa96b builder: Add builder_cache_get_files() to get all files in cache 2017-02-22 09:41:40 +01:00
Alexander Larsson 082efe8951 Properly quote all commandlines we print
Fixes https://github.com/flatpak/flatpak/issues/236
2017-02-21 18:56:11 +01:00
Alexander Larsson 608374e57a builder: Checksum cleanup-platform for platform 2017-02-21 15:09:50 +01:00
Alexander Larsson 1ca3c86552 builder: Add support for --from-git=URL
This allows you to build a json manifest that is stored in a git repo.

Closes: https://github.com/flatpak/flatpak/issues/511
2017-02-21 11:58:40 +01:00
Alexander Larsson b136e7fdc6 builder: extract git mirroring code to builder-git.c
This way we can resuse it outside the git source
2017-02-21 11:58:35 +01:00
Alexander Larsson e71e4221a9 builder: Separate run-dir and base-dir
We always put the state in the .flatpak-builder subdirectory of where
you ran flatpak-builder, independent on the location of the json
manifest. This makes sense in general, but is also extra important
when we want to check out the json from git inside the .flatpak-buidler
directory.
2017-02-21 11:50:58 +01:00
Alexander Larsson 1cdd9e796d builder: Break out build-dir allocation code 2017-02-21 11:50:41 +01:00
Mathieu Bridon b996a636ff builder: Add a "simple" buildsystem
This just runs the specified "build-commands" one after the other,
ignoring makefiles, configure scripts, and all the rest.

Relates to #134
2017-02-20 16:08:35 +01:00
Alexander Larsson c4da82341b builder: Add --build-shell=MODULE support
This lets you prepare a build directory for a given module
and start a build shell inside it. Very useful for debugging.
2017-02-20 15:02:09 +01:00
Alexander Larsson 75a16555e7 builder: Fix spelling 2017-02-17 14:10:12 +01:00
Alexander Larsson be9e8c6eeb builder: Fix unused variable warning 2017-02-16 15:59:01 +01:00
Alexander Larsson bcbd92f774 builder: Bump checksum version to invalidate all caches
Since we changed how the post process handles python
files in the init stage we need to rebuild everything.
2017-02-16 15:52:25 +01:00
Alexander Larsson a5e1b2200a builder: Move python timestamp checkin to post-process
This changes what files we look at to only those in this module,
which is generally right, but to handle the base-layer sdk
case we also have to run the python fixup in the initial layer.
2017-02-16 10:51:00 +01:00
Alexander Larsson ec1589e9ca builder: Fix up get_outstanding changes for first cache stage 2017-02-16 10:27:23 +01:00
Alexander Larsson 05676536a8 builder: Split out debuginfo handling to a post-process step
We want to call this on the initial stage too, so move
it out of BuilderModule.
2017-02-16 09:40:08 +01:00
Alexander Larsson 690908eade builder: Spawn rofiles-fuse with maximum nr of open fds allowed 2017-02-15 20:18:13 +01:00
Alexander Larsson 5a3c7feb15 builder: Unbreak hardlinks when eu-stripping
Otherwise this breaks with rofiles-fuse if the build produces
hardlinked installed files. For instance, as done by mesa.
2017-02-15 19:22:14 +01:00
Alexander Larsson ee83364a8e builder: Work around g_file_replace truncating hardlinked dests
If the target is the rofiles-fuse and a file is hardlinked, then
copy will fall back to truncation, which will fail with READONLY.
We work around this sometimes by deleting the destination first
and sometimes by using g_file_set_contents instead.
2017-02-15 18:04:00 +01:00
Alexander Larsson 1e2e718fd4 builder: Reuse devino cache for builder_cache_get_outstanding_changes
This makes finding debuginfo files to strip much faster.
2017-02-15 17:36:26 +01:00
Alexander Larsson 18b8c67207 builder: Use rofiles-fuse for the cache
Instead of building directly into the app directory we build into a
rofiles-fuse mount of it, which allows us to safely check out the
cache into the app directory using hardlinks (because rofiles-fuse
will not let you modify hardlinked files).

Additionally, every time we commit to the cache we check out all
the new and modified files into the appdir so that we get hardlinks
to the repo for the new files too.

The advantage of having hardlinks to the repo is that we can commit
much more efficient since we don't have to do a full checksum of
the hardlinked files.

There are some issues here:
 eu-strip fails due to doing in-place editin
 rofiles-fuse is using lots of CPU, unclear if this is faster, needs
 measurements
 needs testing of how well the fallback works (ie. if fuse is not
 working).

rofuse: use kernel caches

rofiles: check out after commit

Use devino cache

Only check out new files from cache after commit
2017-02-15 16:09:48 +01:00
Mario Sanchez Prada df680dc0a6 Fix clang build: remove unused variables 2017-02-14 09:00:57 +01:00
Alexander Larsson f2fb0eaf55 By default, don't rebuild apps if the SDK changes
Rebuilding all apps because a minor change in the runtime is way too wasteful
and generated unnecessary app updates, especially since runtimes are supposed
to be API stable.

We add an a --rebuild-on-sdk-change option which you use to disable this feature,
for instance if you're building against an unstable SDK.
2017-02-10 16:37:33 +01:00
Alexander Larsson 08ea3baf49 builder: Download files and archives directly to disk, don't keep them in memory.
This works much better for larger sources.
2017-02-09 12:26:05 +01:00
Alexander Larsson bf316c1a82 builder: Compute SHA256 while downloading archives 2017-02-09 12:17:10 +01:00
Alexander Larsson de2f222474 builder: Don't modify .pyc files in-place
If we need to edit the files, make a copy and edit that.
Otherwise we can't handle the file being hardlinked into
the cache.
2017-02-07 16:39:23 +01:00
Alexander Larsson b585e4008e builder: Store the context in the BuilderCache
This allows the cache to respect global options
2017-02-07 16:39:23 +01:00
Alexander Larsson 4190458d8a Make flatpak-builder run not forward wrong build-finish flags
Some of the build-finish flags we may use make no sense
to pass to flatpak build, so skip them
2017-02-07 16:39:23 +01:00
Alexander Larsson b24228277d Support appdata in share/metainfo also with rename-appdata-file 2017-02-06 11:59:03 +01:00
Jan Grulich 7c7c64c378 Search also in share/metainfo subdir for appstream metadata 2017-02-06 11:54:15 +01:00
Alexander Larsson 69235eeb77 builder: Support cleanup-platform-commands 2017-02-06 10:22:09 +01:00
Alexander Larsson 4feb82e75b builder: Remove the sdk locale/debug extension from the platform
Without this the platform metadata has locale extensions from
both the sdk and the platform
2017-02-02 13:55:30 +01:00
Alexander Larsson e38d980bca builder: Initial version of extension building support 2017-01-27 15:27:55 +01:00
Alexander Larsson 7d4ca72cba builder: User the new --type=runtime to build runtimes 2017-01-27 10:07:23 +01:00
Bartłomiej Piotrowski 751cae4639 builder: add cmake-ninja support
The commit adds another buildsystem, cmake-ninja, that tells
CMake to generate Ninja rules instead of Makefile.

Fixes #461.
2017-01-20 15:28:27 +01:00
Ikey Doherty 605c7ee87b Fix compiler warnings with uninitialised cleanup pointers
This change fixes GCC compiler warnings where a cleanup function was
specified on an uninitialised pointer, which in the view of GCC could
result in an errornous free of uninitialised memory, if the functions
that initialise them do not return NULL.

Signed-off-by: Ikey Doherty <ikey@solus-project.com>
2017-01-19 10:05:13 +01:00
Patrick Griffis a410e86fff builder: Error when failing to include modules
Currently it can silently fail if an included file is invalid
or missing. This makes that fatal.
2016-12-22 14:39:41 +01:00
Alexander Larsson 8737153ded builder: Fix --verbose logging
We need to pass the right log domain in.
2016-12-19 14:55:22 +01:00
Alexander Larsson 364def2135 builder: Add some spew when applying patches
This makes it easier to figure out what patch is broken
2016-12-12 17:20:36 +01:00
Alexander Larsson e514dc2e14 Add flatpak_create_soup_session 2016-12-08 16:41:20 +01:00
Patrick Griffis 21b65947e8 builder: Add buildsystem option and meson support
Closes #400
2016-11-30 16:59:20 +01:00
Alexander Larsson 632f714509 builder: Add "options" property to patch sources
This lets you specify custom arguments to the patch command.

Fixes https://github.com/flatpak/flatpak/issues/417
2016-11-30 10:17:15 +01:00
Alexander Larsson 8ce04c6788 builder: Add support for --jobs=N to limit parallel builds
As requested in: https://github.com/flatpak/flatpak/issues/419
2016-11-30 09:57:55 +01:00
Alexander Larsson 462178bcd3 builder-options: Fix leaks 2016-11-28 17:23:38 +01:00
Alexander Larsson 100038513d builder: Fix coverity warning
If type is NULL we should never look at it.
2016-11-21 10:28:13 +01:00
Matthew Leeds 7b8becd1b2 builder: Don't download modules that won't be built
Currently flatpak-builder downloads the sources for all modules, even if
the --stop-at option prevents them from being built. This commit changes
the behavior so that we don't bother to download the ignored modules.
2016-11-15 15:01:05 +01:00
Alexander Larsson 7c0ef5dccb builder-source: Check for type == NULL before dereferencing 2016-11-11 10:48:37 +01:00
Matthew Leeds 8b50648ec0 builder: Check if .gitmodules exists before showing it. Fixes #366
This commit checks if <revision>:.gitmodules exists in the repo before
calling git-show on it, so an error message doesn't show up in the
program's output.
2016-11-01 09:30:31 +01:00
Alexander Larsson 6e31932221 OCI: Generate an oci image dir structure instead of a runtime 2016-10-31 20:11:14 +01:00
Alexander Larsson 1b09719298 builder: Handle app ids with dashes in them
An app id like org.domain.App-name is now allowed (dashes in the
last element), but blindly appending e.g. ".Locale" to these
for the locale extension will cause an invalid id.

In order to handle this we replace "-" with "_" in this case, which
should be fine as both are under the org.domain control which can
make sure there are no conflicts, and there is no requirement for
the id prefixes to exactly match.

This fixes https://github.com/flatpak/flatpak/issues/370
2016-10-26 21:40:49 +02:00
Alexander Larsson fad8af2f36 builder: Break out name generation for Debug and Locale extensions
These helpers lets us tweak these names
2016-10-26 20:57:57 +02:00
Alexander Larsson 0f05143227 builder: Handle the case where the source directory is in a symlink
It can happen, like in https://github.com/flatpak/flatpak/issues/353
that there are references in the source to the absolute source path.
We need to ensure that is visible in the build. We mostly do this,
but for technical reasons it didn't get the same pathname when
the source was inside a symlinked directory. We fix this with an
extra bind-mount to the symlinked directory name too.

This fixes https://github.com/flatpak/flatpak/issues/353
2016-10-18 11:06:43 +02:00
Alexander Larsson 4cb3f7fd79 builder: Support application inheritance
When initializing a build you can initialize the application directory
with the files from another directory, and optionally a set of its
extensions. This allows one to create a "base" application that
can be derived from in multiple forms. A typical example is an application
framework like Electron. The base electron app has all the dependencies
needed to write Electron app, and then your manifest can add the specific
files needed for your application.

An example snipped could look something like:

    "base": "org.base.App",
    "base-version": "stable",
    "base-extensions": ["org.base.app.Locale", "org.base.app.Debug" ],
2016-10-17 11:40:57 +02:00
Alexander Larsson a9c924928c builder: Add runtime-commit to resolved manifest
Apparently this must have been forgotten when sdk-commit was added.
2016-10-17 10:19:36 +02:00
Matt Watson b9521e719b builder: add allow missing runtimes options
The --allow-missing-runtimes options will allow flatpak builder to
not abort immediately if the sdk or runtime for the app being built
are missing.

This option will be useless when building anything in the modules
section of the app manifest. The calls to flatpak-build will fail
because of the missing sdk.

However, it may be useful when an application does not require
building anything inside the sandbox, and the application files
will be installed via other means.
2016-10-17 09:49:34 +02:00
Matt Watson e968b87f6a builder: add a build finish option
With the option you can resume an application build where it left
off when calling flatpak-buidler --build-only
2016-10-17 09:49:34 +02:00
Alexander Larsson c0e0fe31ef utils: Remove unused variable 2016-10-06 12:31:54 +02:00
Alexander Larsson ea17c27b37 builder: Add missing debug sprew when running an app 2016-10-06 12:31:54 +02:00
Niv Sardi 7f036570fd flatpak-builder: don't strip zip files
It is allowed to make a self-executable zip, it will then be an ELF file
on Linux. The problem is that the current stripping code somehow breaks
that file (i.e. ./file.zip doesn't work anymore after stripping).

Signed-off-by: Niv Sardi <xaiki@endlessm.com>
2016-10-06 12:05:30 +02:00
Alexander Larsson 91c9d474dd flatpak-builder: Add support for --show-deps 2016-09-20 15:58:43 +02:00
Tobias Mueller 72de257ec0 builder: check for name being defined in expand_modules
If this is not checked, flatpak-builder might crash like this:

Starting program: /var/tmp/flatpak/bin/flatpak-builder -v --force-clean
-v --ccache --repo=/tmp/fb.repo --gpg-sign=tobiasmue@gnome.org
/tmp/fpbuilder org.gnome.Sdk.ASan.json
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-
gnu/libthread_db.so.1".
[New Thread 0x7ffff139c700 (LWP 20407)]

Thread 1 "flatpak-builder" received signal SIGSEGV, Segmentation fault.
0x00007ffff697c770 in g_str_hash () from /lib/x86_64-linux-
gnu/libglib-2.0.so.0
(gdb) bt
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    expanded=expanded@entry=0x7fffffffdae0, names=names@entry=0x690a40,
    error=error@entry=0x7fffffffdc08) at builder/builder-manifest.c:175
    expanded=expanded@entry=0x696158, names=names@entry=0x690a40,
    error=error@entry=0x7fffffffdc08) at builder/builder-manifest.c:169
(self=self@entry=0x696050,
    context=context@entry=0x697970, error=error@entry=0x7fffffffdc08)
    at builder/builder-manifest.c:933
    at builder/builder-main.c:349
(gdb) up
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
(gdb)
    expanded=expanded@entry=0x7fffffffdae0, names=names@entry=0x690a40,
    error=error@entry=0x7fffffffdc08) at builder/builder-manifest.c:175
175	      if (g_hash_table_lookup (names, name) != NULL)
(gdb) l
170	        return FALSE;
171
172	      *expanded = g_list_concat (*expanded, submodules);
173
174	      name = builder_module_get_name (m);
175	      if (g_hash_table_lookup (names, name) != NULL)
176	        {
177	          g_set_error (error, G_IO_ERROR, G_IO_ERROR_FAILED,
178	                       "Duplicate modules named '%s'", name);
179	          return FALSE;
(gdb) p names
$1 = (GHashTable *) 0x690a40
(gdb) p name
$2 = 0x0
(gdb) p m
$3 = (BuilderModule *) 0x6978d0
(gdb) p *m
$4 = {parent = {g_type_instance = {g_class = 0x6959d0}, ref_count = 1,
    qdata = 0x0}, name = 0x0, subdir = 0x0, post_install = 0x0,
config_opts = 0x0, make_args = 0x0, make_install_args = 0x0, disabled
= 0,
rm_configure = 0, no_autogen = 0, no_parallel_make = 0,
no_python_timestamp_fix = 0, cmake = 0, builddir = 0, build_options =
0x0,
changes = 0x0, cleanup = 0x0, cleanup_platform = 0x0, sources = 0x0,
modules = 0x0}
(gdb)

This happens when a definition does not include a name, e.g.

            "modules": [
                {
                    "type": "archive",
                    "url": ""foo",
                    "sha256": "123"
                }
2016-09-14 15:52:14 +02:00
Mathieu Bridon d165475e17 builder: Don't fail when submodule path isn't a gitlink
This is what git itself does, so we just should do the same.

cda1bbd474/builtin/submodule--helper.c (L247-L248)
2016-09-13 00:24:42 +02:00
Mathieu Bridon 5430d4d40e builder: Properly check for gitlink trees
Instead of expecting the second word to be "commit", we should do like
git does and check for the file mode, which is 160000 for gitlink trees.

cda1bbd474/cache.h (L96)
2016-09-13 00:24:42 +02:00
Alexander Larsson fba645a999 Remove unused local variables reported by clang 2016-09-12 11:11:35 +02:00
Alexander Larsson 5ef43f6ee3 builder: Bump manifest version to trigger rebuild with new mtimes
To make sure we're not using old caches with the previous mtime
timestamps encoded we bump the manifest version to force a rebuild.
2016-09-09 09:44:54 +02:00
Alexander Larsson 6689c5c7f1 Switch back to mtime==0 for ostree checkouts
OSTree upstream changed back from mtime 1
2016-09-09 09:22:56 +02:00
Alexander Larsson 41443313ca session-helper: Move HostCommand to its own interface 2016-09-06 15:30:38 +02:00
Alexander Larsson e792686892 builder: Propagate build-args to shell sources
This way you can e.g. do network i/o in them.
Thats not a recommended way to do it though, as you work around
a lot of the feature in flatpak-builder like the caching and
verification.
2016-09-05 22:01:56 +02:00
Alexander Larsson c66e394266 builder: Use HostCommand calls if in sandbox 2016-09-05 21:41:14 +02:00
Alexander Larsson ddf05ef063 builder: Use flatpak_spawnv helper in a few more places
This is in preparation for it calling the new host-command.
2016-09-05 21:41:14 +02:00
Alexander Larsson 509bb33a2b Merge pull request #237 from mwleeds/add-trace-msg
utils: Print a debug message when launching subprocesses
2016-08-30 09:22:02 +02:00
Alexander Larsson 07fa8138a2 Merge pull request #194 from smcv/shell-injection
unrpm: prevent shell injection
2016-08-26 11:19:34 +02:00
Alexander Larsson 90d4f8d1b4 Don't fail if appstream-compose fails, instead warn.
This means we stop breaking existing builds e.g. when an older
appstream-compose in the runtime can't handle a new appdata field.
2016-08-25 11:34:10 +02:00
Mathieu Bridon 63cddfa80e builder: Handle submodules using with relative URLs
Some projects declare their Git submodules with URLs relative to their
own.

For example, gedit is hosted at git://git.gnome.org/gedit, and has its
libgd submodule declared with the ../libgd URL. We should treat the
latter as git://git.gnome.org/libgd.

This was in fact properly done in the past, but I broke it in commit
1171e1d202.

This fixes it again.
2016-08-24 21:06:00 +02:00
Alexander Larsson b0a7f8052b builder: Add --sandbox support
This disables build-args support, which is nice when building things
on e.g. a shared build-machine, where we don't want the build to
be able to break out of the sandbox (by specifying e.g. --share=network).
2016-08-22 17:23:32 +02:00