2016-11-27 12:35:58 -08:00
|
|
|
|
/*
|
|
|
|
|
This function composes the Nix Packages collection. It:
|
|
|
|
|
|
2021-01-22 16:36:06 -05:00
|
|
|
|
1. Elaborates `localSystem` and `crossSystem` with defaults as needed.
|
2016-11-27 12:35:58 -08:00
|
|
|
|
|
2021-01-22 16:36:06 -05:00
|
|
|
|
2. Applies the final stage to the given `config` if it is a function
|
2016-11-27 12:35:58 -08:00
|
|
|
|
|
|
|
|
|
3. Defaults to no non-standard config and no cross-compilation target
|
|
|
|
|
|
2016-12-16 05:22:02 -08:00
|
|
|
|
4. Uses the above to infer the default standard environment's (stdenv's)
|
|
|
|
|
stages if no stdenv's are provided
|
2016-11-27 12:35:58 -08:00
|
|
|
|
|
2016-12-16 05:22:02 -08:00
|
|
|
|
5. Folds the stages to yield the final fully booted package set for the
|
|
|
|
|
chosen stdenv
|
2016-11-27 12:35:58 -08:00
|
|
|
|
|
|
|
|
|
Use `impure.nix` to also infer the `system` based on the one on which
|
|
|
|
|
evaluation is taking place, and the configuration from environment variables
|
|
|
|
|
or dot-files.
|
|
|
|
|
*/
|
2016-03-20 16:28:18 +00:00
|
|
|
|
|
2017-02-08 16:43:52 -05:00
|
|
|
|
{
|
|
|
|
|
# The system packages will be built on. See the manual for the
|
|
|
|
|
# subtle division of labor between these two `*System`s and the three
|
|
|
|
|
# `*Platform`s.
|
|
|
|
|
localSystem,
|
|
|
|
|
|
2018-12-04 21:06:46 -06:00
|
|
|
|
# The system packages will ultimately be run on.
|
|
|
|
|
crossSystem ? localSystem,
|
2016-03-20 16:28:18 +00:00
|
|
|
|
|
2016-04-26 10:53:31 -07:00
|
|
|
|
# Allow a configuration attribute set to be passed in as an argument.
|
|
|
|
|
config ? { },
|
2016-03-20 16:28:18 +00:00
|
|
|
|
|
release: forbid use of `lib.fileset` in Nixpkgs
Due to Nix bug <https://github.com/NixOS/nix/issues/11503>,
`builtins.filterSource` and chroot stores interact in a confusing
and broken way that breaks `lib.fileset`. This means that uses of
the API inside Nixpkgs keep breaking the NixOS installer, blocking
the channel. The resulting error messages are inscrutable (they look
like “the installer test is trying to download `curl`…?” and
eventually bottom out in a derivation that has the wrong `outPath`
because of the chroot store causing an incorrect `lib.fileset` result).
Whenever this happens, someone (well, in practice K900 or I)
has to bisect the change that introduced it and remove the use of
`lib.fileset`. This has happened at least three times in the past
four months (I believe I might actually be missing one here, but
these are the ones I remember and could easily dig up):
* <https://github.com/NixOS/nixpkgs/pull/340046>
* <https://github.com/NixOS/nixpkgs/pull/352491>
* <https://github.com/NixOS/nixpkgs/pull/369459>
The options I see here are:
1. Forbid use of `lib.fileset` within Nixpkgs until the Nix bug is
fixed. This is the approach taken here. External users of Nixpkgs
can continue to use the API as normal, but using it from within
something that affects any release jobset `outPath`s will cause an
evaluation failure with a hopefully‐helpful error message.
2. Forbid `lib.fileset` and also all of the other library APIs that use
`builtins.filterSource`. I’m happy to do this, but so far none of
those have broken the installer, so I decided to start small and
worry about the others if they end up causing a problem in practice.
3. Forbid `builtins.filterSource` directly. This is hard and would
require more invasive `builtins.scopedImport` crimes to do at
evaluation time. I think this would realistically have to be done in
something like nixpkgs-vet instead and I didn’t have much luck
shoehorning a check like this into that codebase when I tried.
4. Fix the Nix bug. This would be great! But also it doesn’t seem to be
happening any time soon, it seems difficult to fix in a way that
doesn’t subtly break compatibility with the previous semantics, and
arguably the fix would need backporting all the way back to 2.3
given our minimum version policy.
5. Do nothing; have people continue to innocuously use `lib.fileset`
throughout Nixpkgs, breaking the installer whenever one of them
sneaks in to that closure, causing the channel to be blocked and
requiring expensive bisections to narrow down the inscrutable test
failure to the package using `lib.fileset`, which then needs moving
back off it. This sucks for the people who keep having to track it
down, holds back important channel bumps, and the criteria for when
it’s okay to use `lib.fileset` are not realistically possible to
teach to all contributors.
I'd be happy to work on (2) as an alternative; (3) would be difficult
and seems like overkill, (4) is not really something I trust myself
to do and wouldn’t address the immediate problem, and (5) isn’t
sustainable. I think that the current approach here is the best
trade‐off for now, as `lib.fileset` seems to be the only prominent
user of the `builtins.filterSource` API that works with full store
paths, exposing it to the Nix bug. It’s unfortunate to lose the
nice API, but since we can’t rely on it to produce correct results
and the channels keep getting blocked as a result, I don’t think
we really have an alternative right now.
2024-12-30 15:36:05 +00:00
|
|
|
|
# Temporary hack to let Nixpkgs forbid internal use of `lib.fileset`
|
|
|
|
|
# until <https://github.com/NixOS/nix/issues/11503> is fixed.
|
|
|
|
|
__allowFileset ? true,
|
|
|
|
|
|
2016-12-17 18:05:21 +00:00
|
|
|
|
# List of overlays layers used to extend Nixpkgs.
|
|
|
|
|
overlays ? [ ],
|
|
|
|
|
|
2018-12-04 21:06:46 -06:00
|
|
|
|
# List of overlays to apply to target packages only.
|
|
|
|
|
crossOverlays ? [ ],
|
|
|
|
|
|
2016-12-16 05:22:02 -08:00
|
|
|
|
# A function booting the final package set for a specific standard
|
2017-02-08 16:43:52 -05:00
|
|
|
|
# environment. See below for the arguments given to that function, the type of
|
|
|
|
|
# list it returns.
|
2016-12-16 05:22:02 -08:00
|
|
|
|
stdenvStages ? import ../stdenv,
|
2020-10-01 11:21:32 -07:00
|
|
|
|
|
|
|
|
|
# Ignore unexpected args.
|
|
|
|
|
...
|
2016-11-02 16:39:18 -04:00
|
|
|
|
}@args:
|
2016-03-20 16:28:18 +00:00
|
|
|
|
|
2016-11-27 12:32:56 -08:00
|
|
|
|
let # Rename the function arguments
|
2019-02-23 00:00:01 +00:00
|
|
|
|
config0 = config;
|
2017-03-23 20:49:28 -04:00
|
|
|
|
crossSystem0 = crossSystem;
|
2016-03-20 16:28:18 +00:00
|
|
|
|
|
2016-11-27 12:32:56 -08:00
|
|
|
|
in
|
|
|
|
|
let
|
release: forbid use of `lib.fileset` in Nixpkgs
Due to Nix bug <https://github.com/NixOS/nix/issues/11503>,
`builtins.filterSource` and chroot stores interact in a confusing
and broken way that breaks `lib.fileset`. This means that uses of
the API inside Nixpkgs keep breaking the NixOS installer, blocking
the channel. The resulting error messages are inscrutable (they look
like “the installer test is trying to download `curl`…?” and
eventually bottom out in a derivation that has the wrong `outPath`
because of the chroot store causing an incorrect `lib.fileset` result).
Whenever this happens, someone (well, in practice K900 or I)
has to bisect the change that introduced it and remove the use of
`lib.fileset`. This has happened at least three times in the past
four months (I believe I might actually be missing one here, but
these are the ones I remember and could easily dig up):
* <https://github.com/NixOS/nixpkgs/pull/340046>
* <https://github.com/NixOS/nixpkgs/pull/352491>
* <https://github.com/NixOS/nixpkgs/pull/369459>
The options I see here are:
1. Forbid use of `lib.fileset` within Nixpkgs until the Nix bug is
fixed. This is the approach taken here. External users of Nixpkgs
can continue to use the API as normal, but using it from within
something that affects any release jobset `outPath`s will cause an
evaluation failure with a hopefully‐helpful error message.
2. Forbid `lib.fileset` and also all of the other library APIs that use
`builtins.filterSource`. I’m happy to do this, but so far none of
those have broken the installer, so I decided to start small and
worry about the others if they end up causing a problem in practice.
3. Forbid `builtins.filterSource` directly. This is hard and would
require more invasive `builtins.scopedImport` crimes to do at
evaluation time. I think this would realistically have to be done in
something like nixpkgs-vet instead and I didn’t have much luck
shoehorning a check like this into that codebase when I tried.
4. Fix the Nix bug. This would be great! But also it doesn’t seem to be
happening any time soon, it seems difficult to fix in a way that
doesn’t subtly break compatibility with the previous semantics, and
arguably the fix would need backporting all the way back to 2.3
given our minimum version policy.
5. Do nothing; have people continue to innocuously use `lib.fileset`
throughout Nixpkgs, breaking the installer whenever one of them
sneaks in to that closure, causing the channel to be blocked and
requiring expensive bisections to narrow down the inscrutable test
failure to the package using `lib.fileset`, which then needs moving
back off it. This sucks for the people who keep having to track it
down, holds back important channel bumps, and the criteria for when
it’s okay to use `lib.fileset` are not realistically possible to
teach to all contributors.
I'd be happy to work on (2) as an alternative; (3) would be difficult
and seems like overkill, (4) is not really something I trust myself
to do and wouldn’t address the immediate problem, and (5) isn’t
sustainable. I think that the current approach here is the best
trade‐off for now, as `lib.fileset` seems to be the only prominent
user of the `builtins.filterSource` API that works with full store
paths, exposing it to the Nix bug. It’s unfortunate to lose the
nice API, but since we can’t rely on it to produce correct results
and the channels keep getting blocked as a result, I don’t think
we really have an alternative right now.
2024-12-30 15:36:05 +00:00
|
|
|
|
pristineLib = import ../../lib;
|
|
|
|
|
|
|
|
|
|
lib =
|
|
|
|
|
if __allowFileset then
|
|
|
|
|
pristineLib
|
|
|
|
|
else
|
|
|
|
|
pristineLib.extend (
|
|
|
|
|
_: _: {
|
|
|
|
|
fileset = abort ''
|
|
|
|
|
|
|
|
|
|
The use of `lib.fileset` is currently forbidden in Nixpkgs due to the
|
|
|
|
|
upstream Nix bug <https://github.com/NixOS/nix/issues/11503>. This
|
|
|
|
|
causes difficult‐to‐debug errors when combined with chroot stores,
|
|
|
|
|
such as in the NixOS installer.
|
|
|
|
|
|
|
|
|
|
For packages that require source to be vendored inside Nixpkgs,
|
|
|
|
|
please use a subdirectory of the package instead.
|
|
|
|
|
'';
|
|
|
|
|
}
|
|
|
|
|
);
|
2016-03-20 16:28:18 +00:00
|
|
|
|
|
2021-12-22 13:05:55 +01:00
|
|
|
|
inherit (lib) throwIfNot;
|
2021-12-22 12:52:06 +01:00
|
|
|
|
|
|
|
|
|
checked =
|
|
|
|
|
throwIfNot (lib.isList overlays) "The overlays argument to nixpkgs must be a list." lib.foldr
|
|
|
|
|
(x: throwIfNot (lib.isFunction x) "All overlays passed to nixpkgs must be functions.")
|
|
|
|
|
(r: r)
|
|
|
|
|
overlays
|
|
|
|
|
throwIfNot
|
|
|
|
|
(lib.isList crossOverlays)
|
|
|
|
|
"The crossOverlays argument to nixpkgs must be a list."
|
|
|
|
|
lib.foldr
|
|
|
|
|
(x: throwIfNot (lib.isFunction x) "All crossOverlays passed to nixpkgs must be functions.")
|
|
|
|
|
(r: r)
|
|
|
|
|
crossOverlays;
|
|
|
|
|
|
2021-01-22 16:36:06 -05:00
|
|
|
|
localSystem = lib.systems.elaborate args.localSystem;
|
|
|
|
|
|
|
|
|
|
# Condition preserves sharing which in turn affects equality.
|
2023-09-12 16:55:32 +03:00
|
|
|
|
#
|
|
|
|
|
# See `lib.systems.equals` documentation for more details.
|
|
|
|
|
#
|
|
|
|
|
# Note that it is generally not possible to compare systems as given in
|
|
|
|
|
# parameters, e.g. if systems are initialized as
|
|
|
|
|
#
|
|
|
|
|
# localSystem = { system = "x86_64-linux"; };
|
|
|
|
|
# crossSystem = { config = "x86_64-unknown-linux-gnu"; };
|
|
|
|
|
#
|
|
|
|
|
# Both systems are semantically equivalent as the same vendor and ABI are
|
|
|
|
|
# inferred from the system double in `localSystem`.
|
2021-01-22 16:36:06 -05:00
|
|
|
|
crossSystem =
|
2023-09-12 16:55:32 +03:00
|
|
|
|
let
|
|
|
|
|
system = lib.systems.elaborate crossSystem0;
|
|
|
|
|
in
|
|
|
|
|
if crossSystem0 == null || lib.systems.equals system localSystem then localSystem else system;
|
2021-01-22 16:36:06 -05:00
|
|
|
|
|
2016-06-22 01:33:53 -07:00
|
|
|
|
# Allow both:
|
|
|
|
|
# { /* the config */ } and
|
|
|
|
|
# { pkgs, ... } : { /* the config */ }
|
2019-02-23 00:00:01 +00:00
|
|
|
|
config1 = if lib.isFunction config0 then config0 { inherit pkgs; } else config0;
|
2016-03-20 16:28:18 +00:00
|
|
|
|
|
2019-02-23 00:00:01 +00:00
|
|
|
|
configEval = lib.evalModules {
|
|
|
|
|
modules = [
|
|
|
|
|
./config.nix
|
|
|
|
|
(
|
|
|
|
|
{ options, ... }:
|
|
|
|
|
{
|
|
|
|
|
_file = "nixpkgs.config";
|
2022-04-30 12:57:50 +02:00
|
|
|
|
config = config1;
|
2019-02-23 00:00:01 +00:00
|
|
|
|
}
|
|
|
|
|
)
|
|
|
|
|
];
|
2023-04-17 19:48:53 +02:00
|
|
|
|
class = "nixpkgsConfig";
|
2019-02-23 00:00:01 +00:00
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
# take all the rest as-is
|
2022-04-30 12:57:50 +02:00
|
|
|
|
config = lib.showWarnings configEval.config.warnings configEval.config;
|
2019-02-23 00:00:01 +00:00
|
|
|
|
|
2016-11-02 16:39:18 -04:00
|
|
|
|
# A few packages make a new package set to draw their dependencies from.
|
2025-02-05 14:29:11 +01:00
|
|
|
|
# (Currently to get a cross tool chain, or forced-i686 package.) Rather than
|
|
|
|
|
# give `all-packages.nix` all the arguments to this function, even ones that
|
|
|
|
|
# don't concern it, we give it this function to "re-call" nixpkgs, inheriting
|
|
|
|
|
# whatever arguments it doesn't explicitly provide. This way,
|
|
|
|
|
# `all-packages.nix` doesn't know more than it needs too.
|
2016-11-02 16:39:18 -04:00
|
|
|
|
#
|
2019-02-03 15:30:13 +00:00
|
|
|
|
# It's OK that `args` doesn't include default arguments from this file:
|
2016-11-26 15:34:13 -08:00
|
|
|
|
# they'll be deterministically inferred. In fact we must *not* include them,
|
|
|
|
|
# because it's important that if some parameter which affects the default is
|
|
|
|
|
# substituted with a different argument, the default is re-inferred.
|
|
|
|
|
#
|
2025-02-05 14:29:11 +01:00
|
|
|
|
# To put this in concrete terms, this function is basically just used today to
|
|
|
|
|
# use package for a different platform for the current platform (namely cross
|
|
|
|
|
# compiling toolchains and 32-bit packages on x86_64). In both those cases we
|
|
|
|
|
# want the provided non-native `localSystem` argument to affect the stdenv
|
|
|
|
|
# chosen.
|
2019-02-23 00:00:00 +00:00
|
|
|
|
#
|
|
|
|
|
# NB!!! This thing gets its `config` argument from `args`, i.e. it's actually
|
|
|
|
|
# `config0`. It is important to keep it to `config0` format (as opposed to the
|
|
|
|
|
# result of `evalModules`, i.e. the `config` variable above) throughout all
|
|
|
|
|
# nixpkgs evaluations since the above function `config0 -> config` implemented
|
|
|
|
|
# via `evalModules` is not idempotent. In other words, if you add `config` to
|
|
|
|
|
# `newArgs`, expect strange very hard to debug errors! (Yes, I'm speaking from
|
|
|
|
|
# experience here.)
|
2025-02-05 14:29:11 +01:00
|
|
|
|
nixpkgsFun = newArgs: import ./. (args // newArgs);
|
2016-11-02 16:39:18 -04:00
|
|
|
|
|
2025-02-22 21:23:34 +01:00
|
|
|
|
# Partially apply some arguments for building bootstrapping stage pkgs
|
2016-11-27 12:37:45 -08:00
|
|
|
|
# sets. Only apply arguments which no stdenv would want to override.
|
2016-11-27 12:35:58 -08:00
|
|
|
|
allPackages =
|
|
|
|
|
newArgs:
|
|
|
|
|
import ./stage.nix (
|
|
|
|
|
{
|
2016-11-27 12:37:45 -08:00
|
|
|
|
inherit lib nixpkgsFun;
|
2016-11-27 12:35:58 -08:00
|
|
|
|
}
|
|
|
|
|
// newArgs
|
|
|
|
|
);
|
|
|
|
|
|
2016-12-16 05:22:02 -08:00
|
|
|
|
boot = import ../stdenv/booter.nix { inherit lib allPackages; };
|
|
|
|
|
|
|
|
|
|
stages = stdenvStages {
|
2018-12-04 21:06:46 -06:00
|
|
|
|
inherit
|
|
|
|
|
lib
|
|
|
|
|
localSystem
|
|
|
|
|
crossSystem
|
|
|
|
|
config
|
|
|
|
|
overlays
|
|
|
|
|
crossOverlays
|
|
|
|
|
;
|
2016-11-27 12:35:58 -08:00
|
|
|
|
};
|
|
|
|
|
|
2016-12-16 05:22:02 -08:00
|
|
|
|
pkgs = boot stages;
|
2016-10-12 15:14:49 -04:00
|
|
|
|
|
2021-12-22 12:52:06 +01:00
|
|
|
|
in
|
|
|
|
|
checked pkgs
|