[BUG] project run is not isolated, i.e., it runs all project configurations instead of the specific one

[SalahAdDin]

Describe the bug

We are implementing multiple projects in our tests:

import { storybookTest } from "@storybook/addon-vitest/vitest-plugin";
import { playwright } from "@vitest/browser-playwright";
import { getViteConfig } from "astro/config";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { coverageConfigDefaults } from "vitest/config";

const dirname
  = typeof __dirname !== "undefined"
    ? __dirname
    : path.dirname(fileURLToPath(import.meta.url));

// TODO: Astro v6 requires Vitest 4.1 to avoid issues
export default getViteConfig({
  build: {
    rollupOptions: {
      output: {
        banner:
          "/* ﷽ - In the name of Allah, the Most Gracious, the Most Merciful */",
      },
    },
  },
  resolve: {
    alias: {
      "@components": path.resolve(dirname, "src/components"),
      "@layouts": path.resolve(dirname, "src/layouts"),
      "@styles": path.resolve(dirname, "src/styles"),
      "@utils": path.resolve(dirname, "src/utils"),
      "@": path.resolve(dirname, "src"),
    },
  },
  test: {
    globals: true,
    css: true,
    clearMocks: true,
    restoreMocks: true,
    passWithNoTests: true,
    coverage: {
      exclude: [...coverageConfigDefaults.exclude],
      provider: "v8",
      reporter: ["html", "verbose"],
      thresholds: {
        branches: 90,
        functions: 95,
        lines: 80,
        statements: 80,
      },
    },
    projects: [
      {
        extends: true,
        test: {
          name: "unit",
          include: ["src/**/*.{test,spec}.{astro,js,jsx,ts,tsx}"],
          exclude: ["tests"],
          environment: "node",
          setupFiles: ["./src/vitest.setup.ts"],
        },
      },
      {
        extends: true,
        plugins: [
          // TODO: https://github.com/storybookjs/storybook/issues/33748
          storybookTest({
            configDir: path.join(dirname, ".storybook"),
          }),
        ],
        test: {
          name: "storybook",
          browser: {
            enabled: true,
            headless: true,
            provider: playwright({}),
            instances: [
              {
                browser: "chromium",
              },
            ],
          },
          setupFiles: [".storybook/vitest.setup.ts"],
        },
      },
    ],
  },
});

As Storybook is not ready for development with Astro, we want to ignore its test and run only the project unitary tests: "test:unit": "vitest --project unit",

So, we run it, and we get the following error:

SB_CORE-SERVER_0007 (MainFileEvaluationError): Storybook couldn't evaluate your .storybook\main.ts file.

Original error:
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '\node_modules\@storybook-astro\framework\src\node' imported from \.storybook\main.ts
    at finalizeResolution (node:internal/modules/esm/resolve:275:11)
    at moduleResolve (node:internal/modules/esm/resolve:866:10)
    at defaultResolve (node:internal/modules/esm/resolve:992:11)
    at nextResolve (node:internal/modules/esm/hooks:785:28)
    at AsyncLoaderHooksOnLoaderHookWorker.resolve (node:internal/modules/esm/hooks:269:30)
    at MessagePort.handleMessage (node:internal/modules/esm/worker:251:24)
    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:843:20)
    at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28)



 ELIFECYCLE  Test failed. See above for more details.
husky - pre-push script failed (code 1)
error: failed to push some refs to 'github.com:SalahAdDin/central-mosque-bogota.git'

This is one of the errors we are expecting to get from Storybook, that's why we are not running its tests.

Why is Vitest parsing all projects' configuration instead the required one?

Reproduction

https://stackblitz.com/edit/github-yyntulzp?file=vite.config.ts&preset=node

System Info

System:
    OS: Windows 11 10.0.26200
    CPU: (32) x64 Intel(R) Core(TM) i9-14900HX
    Memory: 37.18 GB / 63.63 GB
  Binaries:
    Node: 24.13.1 - C:\Users\alagu\AppData\Local\fnm_multishells\3408_1773259639075\node.EXE
    npm: 11.8.0 - C:\Users\alagu\AppData\Local\fnm_multishells\3408_1773259639075\npm.CMD
    pnpm: 10.30.1 - C:\Users\alagu\AppData\Local\fnm_multishells\3408_1773259639075\pnpm.CMD
  Browsers:
    Edge: Chromium (140.0.3485.54)
    Internet Explorer: 11.0.26100.7309
  npmPackages:
    @vitest/browser-playwright: 4.0.18 => 4.0.18
    @vitest/coverage-v8: 4.0.18 => 4.0.18
    @vitest/eslint-plugin: 1.6.10 => 1.6.10
    @vitest/ui: 4.0.18 => 4.0.18
    vitest: 4.1.0-beta.6 => 4.1.0-beta.6
    vitest-preview: 0.0.3 => 0.0.3

Used Package Manager

pnpm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[AriPerkkio]

Based on the discussion on Discord earlier, basically what this issue is about is:

import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    projects: [
      {
        plugins: [{ name: 'plugin:one', config() { console.log("Config one") } }],
        test: { name: "one" },
      },
      {
        plugins: [{ name: 'plugin:two', config() { console.log("Config two") } }],
        test: { name: "two" },
      },
    ],
  },
});

$ vitest --project one

Config one
Config two

User does not expect Config two to be logged at all, since --project one filters it out.

The project filtering should happen before config resolution plugin initialization. Of course this would be impossible in cases like:

import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    projects: [
      {
        plugins: [
          {
            name: "plugin:one",
            config() {
              console.log("Config one");
            },
          },
        ],
        test: { name: "one" },
      },
      {
        plugins: [{ name: "plugin:two", config() { return { test: { name: "two" } } } }],
      },
    ],
  },
});

[sheremet-va]

That is indeed impossible because the config hook can influence the name of the project. It is expected behavior that all configuration is resolved before the projects are filtered out.

[SalahAdDin]

That is indeed impossible because the config hook can influence the name of the project. It is expected behavior that all configuration is resolved before the projects are filtered out.

Why that?

[sheremet-va]

Why that?

Because config hook can change the config, including the name. We need to resolve the name before we can filter it.

[sheremet-va]

Closing this as expected behavior. It is a valid use case to use plugins as "presets" and change any configuration value, including the name. Vitest always runs these hooks in all projects: apply, config, configResolved. As a note, vitest doesn't run config hooks if apply is false or a function returning false