Skip to content
This repository was archived by the owner on Aug 11, 2022. It is now read-only.

WIP multi stage install#6191

Closed
iarna wants to merge 14 commits intonpm:masterfrom
iarna:multi-stage
Closed

WIP multi stage install#6191
iarna wants to merge 14 commits intonpm:masterfrom
iarna:multi-stage

Conversation

@iarna
Copy link
Contributor

@iarna iarna commented Sep 15, 2014

  • Load existing tree with read-package-tree
  • Isolate and extract package metadata fetching
  • Recursively determine dependencies
    • From package.json
    • From arguments
    • From developer dependencies
  • Account for shrinkwrap in dependency loader
    • Replace deps at the top level
    • Include similar processing for children too
  • (partially complete) Dedup tree
    • (partially complete) Implement the current strategy, merging back to the lowest common ancestor
      • A version of this exists, but suffers from inexplicable differences from run to run, despite containing no async code. PLUS, adding accesses to certain objects seem to make the differences go away. I put this aside while I work further down the list and'll come back to it with a clear head later.
    • Implement a "as flat as possible" strategy where we merge back to the root when possible
  • Validate the tree
    • Existing tree validations
    • Peer depenedencies
  • Compare the trees and generate a list of high level actions (ie, install, update, remove)
  • Optimize the action list (that is, fold remove+install into update or move as appropriate)
  • Decompose the action list into the lowlevel actions (untar, extract, gyp, lifecycle scripts, etc)
  • Sort the action list
  • Possibly further optimize the action list
  • Execute the actions
    • add locking to these where appropriate
    • fetch
    • extract
    • preinstall
    • build
    • link binaries
    • install (the install lifecycle)
    • postinstall
    • test (w/ npat)
    • remove
    • finalize (move from staging to final location)
  • Update top-level package.json and shrinkwrap when --save is used.
  • Emit wrapped error messages like the old npm-install
  • Load bundled dependenci
  • Add progress bar to replace spinner
    • Integrate progress bar code into npm-log
    • Make progress bar prettier
    • Integrate progress bar into multi-stage install
    • Deeper integration of progress bar into other related tasks, eg, cache, request, rimraf, etc.
  • Implement linked dependency semantics
  • Update this here list to reference the tickets the various bits are addressing
  • Make an external tool to prototype submitting ideal trees to nodesecurity for validation. (Consider as a prototype for potential inclusion as an npm command.) http://blog.nodesecurity.io/2014/02/01/new-feature-validate-modules-with-npm-shrinkwrap
    This will ultimately fix multi-stage install #5919 et al.

Builds now occur in node_modules/.staging and modules are only moved to their respective locations at the very end of the process.

See also the design doc on install semantics.

@iarna iarna force-pushed the multi-stage branch 4 times, most recently from 8706d4e to 3a77511 Compare September 18, 2014 23:08
@iarna iarna force-pushed the multi-stage branch 5 times, most recently from cec4d0c to 8bd789f Compare October 6, 2014 18:09
@iarna iarna added this to the multi-stage install milestone Oct 13, 2014
@iarna iarna force-pushed the multi-stage branch 2 times, most recently from 21a8589 to 873087c Compare October 18, 2014 07:47
@othiym23 othiym23 mentioned this pull request Oct 18, 2014
Closed
@othiym23 othiym23 mentioned this pull request Oct 21, 2014
Closed
@iarna iarna force-pushed the multi-stage branch 8 times, most recently from a6ec4bc to 53846f1 Compare October 27, 2014 05:17
@othiym23 othiym23 mentioned this pull request Oct 27, 2014
Closed
@iarna iarna force-pushed the multi-stage branch 3 times, most recently from f2fa4d2 to 922d758 Compare October 29, 2014 09:46
@iarna iarna force-pushed the multi-stage branch 2 times, most recently from c71632c to e2278bc Compare November 25, 2014 00:38
@iarna iarna force-pushed the multi-stage branch 4 times, most recently from 7800840 to fec7c37 Compare December 9, 2014 03:55
@iarna
Copy link
Contributor Author

iarna commented Dec 15, 2014

Closing this as its tasks have been split into issues and we won't be landing it via pull request anyway.

@iarna iarna closed this Dec 15, 2014
@iarna iarna removed the in-progress label Dec 15, 2014
@Fishrock123
Copy link
Contributor

Fishrock123 commented Jan 7, 2015

Is there a good way to track the progress of this?

@prettycode
Copy link

prettycode commented Mar 5, 2015

I'm also looking for the progress. I'm on Windows and running into the "path too long" problems of nested node_module dependencies. My understanding was that dedup in its current form does not solve this problem and that "multi-stage-install" was going to solve this. Where are we at now with this?

@Fishrock123
Copy link
Contributor

Fishrock123 commented Mar 20, 2015

Looks like it's now on this branch: https://github.com/npm/npm/commits/multi-stage

@othiym23
Copy link
Contributor

othiym23 commented Mar 21, 2015

@Fishrock123 @prettycode Yeah, sorry for not responding earlier. There's a multi-stage branch which represents the current state of the code, and sometime in the next few days @iarna is going to refresh the multi-stage install milestone to look less like a wish list and more like a snapshot of the current issues with the code base. The good news is that this is a pretty thorough revision of the npm installation process, and is pretty much entirely superior to what we had before. The bad news is that it's a pretty thorough revision of the npm installation process and we keep finding new things that need thorough reworking to work with the new code.

@orangemocha
Copy link
Contributor

orangemocha commented Jun 4, 2015

Any updates on when this work will make it into master / an official release?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

@iarna iarna

Labels

None yet

Projects

None yet

Milestone

3.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@iarna @Fishrock123 @prettycode @othiym23 @orangemocha