In [a different PR](https://github.com/vercel/vercel/pull/9009), detecting frameworks by package name will also provide framework version metadata to the build. Should we update these framework detectors to look up their respective packages or were they not doing that already for a reason?
I left the old detectors in place as fallbacks, which looks like:
```
some: [
{
path: 'package.json',
matchContent:
'"(dev)?(d|D)ependencies":\\s*{[^}]*"remix":\\s*".+?"[^}]*}',
},
{
path: 'remix.config.js',
},
],
```
Please review carefully.
This PR:
- updates `packages/frameworks` to have most supported frameworks specify which dependency version should reflect the overall framework version
- updates `packages/fs-detectors` to allow framework detection that returns the full `Framework` record instead of just the slug
- updates `packages/next` to return the detected Next.js version in the build result
- updates `packages/cli` to leverage these changes so that `vc build` can add `framework: { version: string; }` to `config.json` output
The result is that Build Output API and supported frameworks will return their framework version in the build result of `vc build` when possible, which is used by the build container when creating the deployment. The dashboard later retrieves this value to display in richer deployment outputs.
Supports:
- https://github.com/vercel/api/pull/15601
- https://github.com/vercel/front/pull/18319
---
With the related build container updates, we get to see Next.js version in the build output. You'll see this with BOA+Prebuilt or a normal deploy:
<img width="1228" alt="Screen Shot 2022-12-09 at 2 48 12 PM" src="https://user-images.githubusercontent.com/41545/206793639-f9cd3bdf-b822-45dd-9564-95b94994271d.png">
---
### The Path to this PR
I went through all the supported frameworks and figured out how to best determine their versions. For most of them, we can check a known dependency's installed version number.
We can get most of the way only checking npm. For a handful, we'd have to support Go/Ruby/Rust/Whatever dependencies.
I started with a more complex method signature to allow for later expansion without changing the signature. It looked like this, in practice:
```
async getVersion(dependencies: DependencyMap) => depedencies['next']
```
However, after checking all currently supported frameworks, I don't think this will end up being necessary. It also has the constraint that all dependencies have to be gathered and presented to the function even though it only needs to check for one or two. That's not a huge deal if we have them already where we need them, but we don't. We could use a variant here where this function does its own lookups, but this seemed unnecessary and would beg for duplication and small variances that could cause bugs.
Further, if we only look at `package.json`, we're going to either see a specific version of a version range. To be precise, we have to look at the installed version of the package. That means checking one of the various types of lockfiles that can exist or poking into node_modules.
If we poke into node_modules to detect the installed version, we introduce another point where Yarn 3 (default mode) will not be supported. If we read lockfiles, we have to potentially parse `npm`, `pnpm`, and `yarn` lockfiles.
If we use `npm ls <package-name>`, that also fails in Yarn 3 (default mode). We could accept that and go forward anyway, which would look like:
```
const args = `ls ${packageName} --depth=0 --json`.split(' ');
const { stdout } = await execa('npm', args, { cwd });
const regex = new RegExp(String.raw`${packageName}@([\.\d]+)`);
const matches = stdout.match(regex);
if (matches) {
return matches[1];
}
```
But it turns out there's a `--json` option! That's what I ended up using, for now.
We could explore the lockfile route more, but after some initial digging, it' non-trivial. There are 3 main lockfiles we'd want to check for (npm, pnpm, and yarn) and there are different lockfile versions that put necessary data in different places. I looked for existing tools that parse this, but I didn't find any. We could certainly go down this path, but the effort doesn't seem worth it when `npm ls` gets us really close.
---
### Follow-up Versioning
Now that we know how to determine version per framework, we can vary configuration by version. In a future PR, we could allow a given value to vary by version number:
```
name: (version) => {
if (semver.gt(version, '9.8.7')) {
return 'some-framework-2''
}
return 'some-framework';
}
```
However, it may still be easier to differentiate significant versions by adding multiple entries in the list.
### Related Issues
Improves how `vc build` handles monorepos. In short, this introduces
monorepo manager detection logic and then some helpful defaults so users
don't have to manually specify a `buildCommand` or `installCommand` when
linking a project within a monorepo.
### 📋 Checklist
<!--
Please keep your PR as a Draft until the checklist is complete
-->
#### Tests
- [ ] The code changed/added as part of this PR has been covered with
tests
- [ ] All tests pass locally with `yarn test-unit`
#### Code Review
- [ ] This PR has a concise title and thorough description useful to a
reviewer
- [ ] Issue from task tracker has a link to this PR
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This is to fix the issue with processing new entries of `potentialFiles` when there was a `readdir` cache hit
### Related Issues
Related to https://linear.app/vercel/issue/HIT-57/monorepo-detection-api-prevent-rate-limits
### 📋 Checklist
<!--
Please keep your PR as a Draft until the checklist is complete
-->
#### Tests
- [ ] The code changed/added as part of this PR has been covered with tests
- [ ] All tests pass locally with `yarn test-unit`
#### Code Review
- [ ] This PR has a concise title and thorough description useful to a reviewer
- [ ] Issue from task tracker has a link to this PR
### Related Issues
https://nx.dev/more-concepts/folder-structure#integrated-repo-folder-structure
Nx monorepo has an option to use Nx workspaces.
Nx workspace is defined within the root `workspace.json` file
Within this `workspace.json` file the workspace packages are under
projects
```{
"$schema": "./node_modules/nx/schemas/workspace-schema.json",
"version": 2,
"projects": {
"myblog": "apps/myblog",
"svelte-app": "apps/svelte-app",
}
}
```
Within `getNxWorkspacePackagePaths` get the projects object values for
the paths
Nx is listed as the last workspace manager within
`workspace-managers.ts` because other workspace managers could exist to
check for before Nx workspaces because the `workspace.json` could exist
but not be the correct workspace manager
Nx workspace file can exist when yarn/npm workspaces exist. When this
happens, the workspace.json file is empty with no projects so it will
not add any package paths to the list to look through for projects.
### 📋 Checklist
<!--
Please keep your PR as a Draft until the checklist is complete
-->
#### Tests
- [ ] The code changed/added as part of this PR has been covered with
tests
- [ ] All tests pass locally with `yarn test-unit`
#### Code Review
- [ ] This PR has a concise title and thorough description useful to a
reviewer
- [ ] Issue from task tracker has a link to this PR
Adds a `writeFile` function to `DetectorFilesystem` that will be used to update the various file cache maps.
**Why is this needed?**
When detecting npm7+ monorepos, we identified a performance improvement where the service can inspect the `package-lock.json` file for workspaces, and reuse the package information for each workspace in framework-detection.
The pseudo code in `vercel/api` will look something like this
For a given lockfile
```json
{
...,
"packages": {
"": {
"name": "npm-workspaces",
"version": "1.0.0",
"license": "ISC",
"workspaces": {
"packages": [
"apps/*"
]
}
},
"apps/admin": {
"version": "0.1.0",
"dependencies": {
"next": "12.2.5",
"react": "18.2.0",
"react-dom": "18.2.0"
},
"devDependencies": {
"eslint": "8.23.0",
"eslint-config-next": "12.2.5"
}
},
...,
}
```
```ts
// for each projectPath we detect in package-lock.json
// switch the cwd of the fs to the project directory
const projectFs = fs.chdir(projectPath);
// gets the package info from the lockfile
const projectPackageInfo = lockFile.packages[projectPath];
// insert this content into fs cache
await projectFs.writeFile('package.json', projectPackageInfo)
// call detectFramework, which should now have a cached "package.json" file
const projectFramework = await detectFramework(projectFs);
```
### Related Issues
Related to https://linear.app/vercel/issue/HIT-57/monorepo-detection-api-prevent-rate-limits
### 📋 Checklist
<!--
Please keep your PR as a Draft until the checklist is complete
-->
#### Tests
- [x] The code changed/added as part of this PR has been covered with tests
- [x] All tests pass locally with `yarn test-unit`
#### Code Review
- [x] This PR has a concise title and thorough description useful to a reviewer
- [x] Issue from task tracker has a link to this PR
Adding the ability to explicitly set the `hasPath` and `isFile` caches via `readdir`
**Why is this needed?**
For framework detection, this allows us to fetch the contents of a directory (the root directory) of a git repo, and make assumptions about the existence of files,
```ts
import frameworks from "@vercel/frameworks";
// we can make the set of all file paths needed for framework detection like this
const setOfPaths = new Set(
frameworks.frameworks
.flatMap((f) => [
...(f.detectors?.every ?? []),
...(f.detectors?.some ?? []),
])
.map((d) => d.path)
);
// then we can read the contents of the root directory of a git repo via the `DetectorFilesystem`
fs.readdir('./', { potentialFiles: [...setOfPaths] });
// The filesystem has been fully primed - any network calls from here will only happen if needed.
// This works because the logic of `detectFramework` calls `hasPath` and then `isFile` before
// fetching file contents. Since the filesystem knows this information, it doesn't have to make
// any additional network calls 🙌
const framework = detectFramework(fs);
```
### Related Issues
Related to [HIT-57](https://linear.app/vercel/issue/HIT-57/monorepo-detection-api-prevent-rate-limits)
### 📋 Checklist
<!--
Please keep your PR as a Draft until the checklist is complete
-->
#### Tests
- [x] The code changed/added as part of this PR has been covered with tests
- [x] All tests pass locally with `yarn test-unit`
#### Code Review
- [x] This PR has a concise title and thorough description useful to a reviewer
- [x] Issue from task tracker has a link to this PR
Adds a `writeFile` function to `DetectorFilesystem` that will be used to update the various file cache maps.
When detecting npm7+ monorepos, we identified a performance improvement where the service can inspect the `package-lock.json` file for workspaces, and reuse the package information for each workspace in framework-detection.
The pseudo code in `vercel/api` will look something like this
For a given lockfile
```json
{
...,
"packages": {
"": {
"name": "npm-workspaces",
"version": "1.0.0",
"license": "ISC",
"workspaces": {
"packages": [
"apps/*"
]
}
},
"apps/admin": {
"version": "0.1.0",
"dependencies": {
"next": "12.2.5",
"react": "18.2.0",
"react-dom": "18.2.0"
},
"devDependencies": {
"eslint": "8.23.0",
"eslint-config-next": "12.2.5"
}
},
...,
}
```
```ts
// for each projectPath we detect in package-lock.json
// switch the cwd of the fs to the project directory
const projectFs = fs.chdir(projectPath);
// gets the package info from the lockfile
const projectPackageInfo = lockFile.packages[projectPath];
// insert this content into fs cache
projectFs.writeFile('package.json', projectPackageInfo)
// call detectFramework, which should now have a cached "package.json" file
const projectFramework = await detectFramework(projectFs);
```
### Related Issues
Related to [HIT-57](https://linear.app/vercel/issue/HIT-57/monorepo-detection-api-prevent-rate-limits)
### 📋 Checklist
<!--
Please keep your PR as a Draft until the checklist is complete
-->
#### Tests
- [x] The code changed/added as part of this PR has been covered with tests
- [x] All tests pass locally with `yarn test-unit`
#### Code Review
- [ ] This PR has a concise title and thorough description useful to a reviewer
- [ ] Issue from task tracker has a link to this PR
The Vercel API's `detect-framework` API contacts GitHub's API using its own file adapter.
However, currently the `detectFramework` function in the Vercel CLI (on which the API depends) calls the file system adapter in series using a `for` loop. This results in large amounts of lag when in a networked situation such as calling GitHub's API as each API call takes a couple hundred milliseconds.
I propose changing `detectFramework` to process the directories it scans in parallel.
### 📋 Checklist
<!--
Please keep your PR as a Draft until the checklist is complete
-->
#### Tests
- [x] The code changed/added as part of this PR has been covered with tests
- [x] All tests pass locally with `yarn test-unit`
#### Code Review
- [x] This PR has a concise title and thorough description useful to a reviewer
- [x] Issue from task tracker has a link to this PR
The `@vercel/build-utils` package was meant be shared functions necessary for writing a Vercel Builder (aka Runtime).
This package has since bloated into the catch-all package for anything that wasn't a Builder.
This PR removes the bloat in favor of a new package, `@vercel/fs-detectors`. It also removes the need for `@vercel/build-utils` to have a dependency on `@vercel/frameworks`.
- Related to #7951
