This commit is contained in:
2022-07-18 02:50:52 +00:00
parent befd344ab0
commit 06181b34d6
8569 changed files with 818704 additions and 352705 deletions
+98 -12
View File
@@ -12,8 +12,6 @@ when running as `root`.
It was written to be used as [npm](https://npm.im)'s local cache, but can
just as easily be used on its own.
_Translations: [español](README.es.md)_
## Install
`$ npm install --save cacache`
@@ -35,10 +33,11 @@ _Translations: [español](README.es.md)_
* Writing
* [`put`](#put-data)
* [`put.stream`](#put-stream)
* [`put*` opts](#put-options)
* [`rm.all`](#rm-all)
* [`rm.entry`](#rm-entry)
* [`rm.content`](#rm-content)
* [`index.compact`](#index-compact)
* [`index.insert`](#index-insert)
* Utilities
* [`clearMemoized`](#clear-memoized)
* [`tmp.mkdir`](#tmp-mkdir)
@@ -94,7 +93,7 @@ cacache.get.byDigest(cachePath, integrityHash).then(data => {
* Lockless, high-concurrency cache access
* Streaming support
* Promise support
* Pretty darn fast -- sub-millisecond reads and writes including verification
* Fast -- sub-millisecond reads and writes including verification
* Arbitrary metadata storage
* Garbage collection and additional offline verification
* Thorough test coverage
@@ -203,6 +202,8 @@ A sub-function, `get.byDigest` may be used for identical behavior, except lookup
will happen by integrity hash, bypassing the index entirely. This version of the
function *only* returns `data` itself, without any wrapper.
See: [options](#get-options)
##### Note
This function loads the entire cache entry into memory before returning it. If
@@ -244,6 +245,8 @@ A sub-function, `get.stream.byDigest` may be used for identical behavior,
except lookup will happen by integrity hash, bypassing the index entirely. This
version does not emit the `metadata` and `integrity` events at all.
See: [options](#get-options)
##### Example
```javascript
@@ -330,12 +333,33 @@ cacache.get.hasContent(cachePath, 'sha521-NOT+IN/CACHE==').then(console.log)
false
```
##### <a name="get-options"></a> Options
##### `opts.integrity`
If present, the pre-calculated digest for the inserted content. If this option
is provided and does not match the post-insertion digest, insertion will fail
with an `EINTEGRITY` error.
##### `opts.memoize`
Default: null
If explicitly truthy, cacache will read from memory and memoize data on bulk read. If `false`, cacache will read from disk data. Reader functions by default read from in-memory cache.
##### `opts.size`
If provided, the data stream will be verified to check that enough data was
passed through. If there's more or less data than expected, insertion will fail
with an `EBADSIZE` error.
#### <a name="put-data"></a> `> cacache.put(cache, key, data, [opts]) -> Promise`
Inserts data passed to it into the cache. The returned Promise resolves with a
digest (generated according to [`opts.algorithms`](#optsalgorithms)) after the
cache entry has been successfully written.
See: [options](#put-options)
##### Example
```javascript
@@ -355,6 +379,8 @@ Stream](https://nodejs.org/api/stream.html#stream_writable_streams) that inserts
data written to it into the cache. Emits an `integrity` event with the digest of
written contents when it succeeds.
See: [options](#put-options)
##### Example
```javascript
@@ -367,9 +393,7 @@ request.get(
)
```
#### <a name="put-options"></a> `> cacache.put options`
`cacache.put` functions have a number of options in common.
##### <a name="put-options"></a> Options
##### `opts.metadata`
@@ -384,7 +408,7 @@ with an `EBADSIZE` error.
##### `opts.integrity`
If present, the pre-calculated digest for the inserted content. If this option
if provided and does not match the post-insertion digest, insertion will fail
is provided and does not match the post-insertion digest, insertion will fail
with an `EINTEGRITY` error.
`algorithms` has no effect if this option is present.
@@ -417,6 +441,11 @@ cache.
Reading from disk data can be forced by explicitly passing `memoize: false` to
the reader functions, but their default will be to read from memory.
##### `opts.tmpPrefix`
Default: null
Prefix to append on the temporary directory name inside the cache's tmp dir.
#### <a name="rm-all"></a> `> cacache.rm.all(cache) -> Promise`
Clears the entire cache. Mainly by blowing away the cache directory itself.
@@ -429,13 +458,17 @@ cacache.rm.all(cachePath).then(() => {
})
```
#### <a name="rm-entry"></a> `> cacache.rm.entry(cache, key) -> Promise`
#### <a name="rm-entry"></a> `> cacache.rm.entry(cache, key, [opts]) -> Promise`
Alias: `cacache.rm`
Removes the index entry for `key`. Content will still be accessible if
requested directly by content address ([`get.stream.byDigest`](#get-stream)).
By default, this appends a new entry to the index with an integrity of `null`.
If `opts.removeFully` is set to `true` then the index file itself will be
physically deleted rather than appending a `null`.
To remove the content itself (which might still be used by other entries), use
[`rm.content`](#rm-content). Or, to safely vacuum any unused content, use
[`verify`](#verify).
@@ -462,6 +495,34 @@ cacache.rm.content(cachePath, 'sha512-SoMeDIGest/IN+BaSE64==').then(() => {
})
```
#### <a name="index-compact"></a> `> cacache.index.compact(cache, key, matchFn, [opts]) -> Promise`
Uses `matchFn`, which must be a synchronous function that accepts two entries
and returns a boolean indicating whether or not the two entries match, to
deduplicate all entries in the cache for the given `key`.
If `opts.validateEntry` is provided, it will be called as a function with the
only parameter being a single index entry. The function must return a Boolean,
if it returns `true` the entry is considered valid and will be kept in the index,
if it returns `false` the entry will be removed from the index.
If `opts.validateEntry` is not provided, however, every entry in the index will
be deduplicated and kept until the first `null` integrity is reached, removing
all entries that were written before the `null`.
The deduplicated list of entries is both written to the index, replacing the
existing content, and returned in the Promise.
#### <a name="index-insert"></a> `> cacache.index.insert(cache, key, integrity, opts) -> Promise`
Writes an index entry to the cache for the given `key` without writing content.
It is assumed if you are using this method, you have already stored the content
some other way and you only wish to add a new index to that content. The `metadata`
and `size` properties are read from `opts` and used as part of the index entry.
Returns a Promise resolving to the newly added entry.
#### <a name="clear-memoized"></a> `> cacache.clearMemoized()`
Completely resets the in-memory entry cache.
@@ -480,6 +541,8 @@ permissions.
If you want automatic cleanup of this directory, use
[`tmp.withTmp()`](#with-tpm)
See: [options](#tmp-options)
##### Example
```javascript
@@ -519,6 +582,8 @@ promise completes.
The same caveats apply when it comes to managing permissions for the tmp dir's
contents.
See: [options](#tmp-options)
##### Example
```javascript
@@ -529,6 +594,13 @@ cacache.tmp.withTmp(cache, dir => {
})
```
##### <a name="tmp-options"></a> Options
##### `opts.tmpPrefix`
Default: null
Prefix to append on the temporary directory name inside the cache's tmp dir.
#### <a name="integrity"></a> Subresource Integrity Digests
For content verification and addressing, cacache uses strings following the
@@ -584,10 +656,24 @@ When it's done, it'll return an object with various stats about the verification
process, including amount of storage reclaimed, number of valid entries, number
of entries removed, etc.
##### Options
##### <a name="verify-options"></a> Options
* `opts.filter` - receives a formatted entry. Return false to remove it.
Note: might be called more than once on the same entry.
##### `opts.concurrency`
Default: 20
Number of concurrently read files in the filesystem while doing clean up.
##### `opts.filter`
Receives a formatted entry. Return false to remove it.
Note: might be called more than once on the same entry.
##### `opts.log`
Custom logger function:
```
log: { silly () {} }
log.silly('verify', 'verifying cache at', cache)
```
##### Example
+143 -172
View File
@@ -1,127 +1,112 @@
'use strict'
const Collect = require('minipass-collect')
const Minipass = require('minipass')
const Pipeline = require('minipass-pipeline')
const fs = require('fs')
const util = require('util')
const figgyPudding = require('figgy-pudding')
const fs = require('fs')
const index = require('./lib/entry-index')
const memo = require('./lib/memoization')
const read = require('./lib/content/read')
const Minipass = require('minipass')
const Collect = require('minipass-collect')
const Pipeline = require('minipass-pipeline')
const writeFile = util.promisify(fs.writeFile)
const GetOpts = figgyPudding({
integrity: {},
memoize: {},
size: {}
})
module.exports = function get (cache, key, opts) {
return getData(false, cache, key, opts)
}
module.exports.byDigest = function getByDigest (cache, digest, opts) {
return getData(true, cache, digest, opts)
}
function getData (byDigest, cache, key, opts) {
opts = GetOpts(opts)
const memoized = byDigest
? memo.get.byDigest(cache, key, opts)
: memo.get(cache, key, opts)
if (memoized && opts.memoize !== false) {
return Promise.resolve(
byDigest
? memoized
: {
metadata: memoized.entry.metadata,
data: memoized.data,
integrity: memoized.entry.integrity,
size: memoized.entry.size
}
)
function getData (cache, key, opts = {}) {
const { integrity, memoize, size } = opts
const memoized = memo.get(cache, key, opts)
if (memoized && memoize !== false) {
return Promise.resolve({
metadata: memoized.entry.metadata,
data: memoized.data,
integrity: memoized.entry.integrity,
size: memoized.entry.size,
})
}
return (byDigest ? Promise.resolve(null) : index.find(cache, key, opts)).then(
(entry) => {
if (!entry && !byDigest) {
throw new index.NotFoundError(cache, key)
return index.find(cache, key, opts).then((entry) => {
if (!entry)
throw new index.NotFoundError(cache, key)
return read(cache, entry.integrity, { integrity, size }).then((data) => {
if (memoize)
memo.put(cache, entry, data, opts)
return {
data,
metadata: entry.metadata,
size: entry.size,
integrity: entry.integrity,
}
return read(cache, byDigest ? key : entry.integrity, {
integrity: opts.integrity,
size: opts.size
})
.then((data) =>
byDigest
? data
: {
metadata: entry.metadata,
data: data,
size: entry.size,
integrity: entry.integrity
}
)
.then((res) => {
if (opts.memoize && byDigest) {
memo.put.byDigest(cache, key, res, opts)
} else if (opts.memoize) {
memo.put(cache, entry, res.data, opts)
}
return res
})
}
)
}
module.exports.sync = function get (cache, key, opts) {
return getDataSync(false, cache, key, opts)
}
module.exports.sync.byDigest = function getByDigest (cache, digest, opts) {
return getDataSync(true, cache, digest, opts)
}
function getDataSync (byDigest, cache, key, opts) {
opts = GetOpts(opts)
const memoized = byDigest
? memo.get.byDigest(cache, key, opts)
: memo.get(cache, key, opts)
if (memoized && opts.memoize !== false) {
return byDigest
? memoized
: {
metadata: memoized.entry.metadata,
data: memoized.data,
integrity: memoized.entry.integrity,
size: memoized.entry.size
}
}
const entry = !byDigest && index.find.sync(cache, key, opts)
if (!entry && !byDigest) {
throw new index.NotFoundError(cache, key)
}
const data = read.sync(cache, byDigest ? key : entry.integrity, {
integrity: opts.integrity,
size: opts.size
})
})
const res = byDigest
? data
: {
metadata: entry.metadata,
data: data,
size: entry.size,
integrity: entry.integrity
}
module.exports = getData
function getDataByDigest (cache, key, opts = {}) {
const { integrity, memoize, size } = opts
const memoized = memo.get.byDigest(cache, key, opts)
if (memoized && memoize !== false)
return Promise.resolve(memoized)
return read(cache, key, { integrity, size }).then((res) => {
if (memoize)
memo.put.byDigest(cache, key, res, opts)
return res
})
}
module.exports.byDigest = getDataByDigest
function getDataSync (cache, key, opts = {}) {
const { integrity, memoize, size } = opts
const memoized = memo.get(cache, key, opts)
if (memoized && memoize !== false) {
return {
metadata: memoized.entry.metadata,
data: memoized.data,
integrity: memoized.entry.integrity,
size: memoized.entry.size,
}
if (opts.memoize && byDigest) {
memo.put.byDigest(cache, key, res, opts)
} else if (opts.memoize) {
memo.put(cache, entry, res.data, opts)
}
const entry = index.find.sync(cache, key, opts)
if (!entry)
throw new index.NotFoundError(cache, key)
const data = read.sync(cache, entry.integrity, {
integrity: integrity,
size: size,
})
const res = {
metadata: entry.metadata,
data: data,
size: entry.size,
integrity: entry.integrity,
}
if (memoize)
memo.put(cache, entry, res.data, opts)
return res
}
module.exports.stream = getStream
module.exports.sync = getDataSync
function getDataByDigestSync (cache, digest, opts = {}) {
const { integrity, memoize, size } = opts
const memoized = memo.get.byDigest(cache, digest, opts)
if (memoized && memoize !== false)
return memoized
const res = read.sync(cache, digest, {
integrity: integrity,
size: size,
})
if (memoize)
memo.put.byDigest(cache, digest, res, opts)
return res
}
module.exports.sync.byDigest = getDataByDigestSync
const getMemoizedStream = (memoized) => {
const stream = new Minipass()
@@ -134,20 +119,19 @@ const getMemoizedStream = (memoized) => {
return stream
}
function getStream (cache, key, opts) {
opts = GetOpts(opts)
function getStream (cache, key, opts = {}) {
const { memoize, size } = opts
const memoized = memo.get(cache, key, opts)
if (memoized && opts.memoize !== false) {
if (memoized && memoize !== false)
return getMemoizedStream(memoized)
}
const stream = new Pipeline()
index
.find(cache, key)
.then((entry) => {
if (!entry) {
if (!entry)
throw new index.NotFoundError(cache, key)
}
stream.emit('metadata', entry.metadata)
stream.emit('integrity', entry.integrity)
stream.emit('size', entry.size)
@@ -160,12 +144,10 @@ function getStream (cache, key, opts) {
const src = read.readStream(
cache,
entry.integrity,
opts.concat({
size: opts.size == null ? entry.size : opts.size
})
{ ...opts, size: typeof size !== 'number' ? entry.size : size }
)
if (opts.memoize) {
if (memoize) {
const memoStream = new Collect.PassThrough()
memoStream.on('collect', data => memo.put(cache, entry, data, opts))
stream.unshift(memoStream)
@@ -177,20 +159,20 @@ function getStream (cache, key, opts) {
return stream
}
module.exports.stream.byDigest = getStreamDigest
module.exports.stream = getStream
function getStreamDigest (cache, integrity, opts) {
opts = GetOpts(opts)
function getStreamDigest (cache, integrity, opts = {}) {
const { memoize } = opts
const memoized = memo.get.byDigest(cache, integrity, opts)
if (memoized && opts.memoize !== false) {
if (memoized && memoize !== false) {
const stream = new Minipass()
stream.end(memoized)
return stream
} else {
const stream = read.readStream(cache, integrity, opts)
if (!opts.memoize) {
if (!memoize)
return stream
}
const memoStream = new Collect.PassThrough()
memoStream.on('collect', data => memo.put.byDigest(
cache,
@@ -202,65 +184,54 @@ function getStreamDigest (cache, integrity, opts) {
}
}
module.exports.stream.byDigest = getStreamDigest
function info (cache, key, opts = {}) {
const { memoize } = opts
const memoized = memo.get(cache, key, opts)
if (memoized && memoize !== false)
return Promise.resolve(memoized.entry)
else
return index.find(cache, key)
}
module.exports.info = info
function info (cache, key, opts) {
opts = GetOpts(opts)
const memoized = memo.get(cache, key, opts)
if (memoized && opts.memoize !== false) {
return Promise.resolve(memoized.entry)
} else {
return index.find(cache, key)
}
}
module.exports.hasContent = read.hasContent
function cp (cache, key, dest, opts) {
return copy(false, cache, key, dest, opts)
}
module.exports.copy = cp
function cpDigest (cache, digest, dest, opts) {
return copy(true, cache, digest, dest, opts)
}
module.exports.copy.byDigest = cpDigest
function copy (byDigest, cache, key, dest, opts) {
opts = GetOpts(opts)
function copy (cache, key, dest, opts = {}) {
if (read.copy) {
return (byDigest
? Promise.resolve(null)
: index.find(cache, key, opts)
).then((entry) => {
if (!entry && !byDigest) {
return index.find(cache, key, opts).then((entry) => {
if (!entry)
throw new index.NotFoundError(cache, key)
}
return read
.copy(cache, byDigest ? key : entry.integrity, dest, opts)
return read.copy(cache, entry.integrity, dest, opts)
.then(() => {
return byDigest
? key
: {
metadata: entry.metadata,
size: entry.size,
integrity: entry.integrity
}
return {
metadata: entry.metadata,
size: entry.size,
integrity: entry.integrity,
}
})
})
}
return getData(byDigest, cache, key, opts).then((res) => {
return writeFile(dest, byDigest ? res : res.data).then(() => {
return byDigest
? key
: {
metadata: res.metadata,
size: res.size,
integrity: res.integrity
}
return getData(cache, key, opts).then((res) => {
return writeFile(dest, res.data).then(() => {
return {
metadata: res.metadata,
size: res.size,
integrity: res.integrity,
}
})
})
}
module.exports.copy = copy
function copyByDigest (cache, key, dest, opts = {}) {
if (read.copy)
return read.copy(cache, key, dest, opts).then(() => key)
return getDataByDigest(cache, key, opts).then((res) => {
return writeFile(dest, res).then(() => key)
})
}
module.exports.copy.byDigest = copyByDigest
module.exports.hasContent = read.hasContent
+5
View File
@@ -7,6 +7,11 @@ const rm = require('./rm.js')
const verify = require('./verify.js')
const { clearMemoized } = require('./lib/memoization.js')
const tmp = require('./lib/util/tmp.js')
const index = require('./lib/entry-index.js')
module.exports.index = {}
module.exports.index.compact = index.compact
module.exports.index.insert = index.insert
module.exports.ls = ls
module.exports.ls.stream = ls.stream
+4 -3
View File
@@ -15,9 +15,10 @@ module.exports = contentPath
function contentPath (cache, integrity) {
const sri = ssri.parse(integrity, { single: true })
// contentPath is the *strongest* algo given
return path.join.apply(
path,
[contentDir(cache), sri.algorithm].concat(hashToSegments(sri.hexDigest()))
return path.join(
contentDir(cache),
sri.algorithm,
...hashToSegments(sri.hexDigest())
)
}
+45 -62
View File
@@ -2,8 +2,7 @@
const util = require('util')
const figgyPudding = require('figgy-pudding')
const fs = require('graceful-fs')
const fs = require('fs')
const fsm = require('fs-minipass')
const ssri = require('ssri')
const contentPath = require('./path')
@@ -12,30 +11,25 @@ const Pipeline = require('minipass-pipeline')
const lstat = util.promisify(fs.lstat)
const readFile = util.promisify(fs.readFile)
const ReadOpts = figgyPudding({
size: {}
})
module.exports = read
const MAX_SINGLE_READ_SIZE = 64 * 1024 * 1024
function read (cache, integrity, opts) {
opts = ReadOpts(opts)
function read (cache, integrity, opts = {}) {
const { size } = opts
return withContentSri(cache, integrity, (cpath, sri) => {
// get size
return lstat(cpath).then(stat => ({ stat, cpath, sri }))
}).then(({ stat, cpath, sri }) => {
if (typeof opts.size === 'number' && stat.size !== opts.size) {
throw sizeError(opts.size, stat.size)
}
if (stat.size > MAX_SINGLE_READ_SIZE) {
if (typeof size === 'number' && stat.size !== size)
throw sizeError(size, stat.size)
if (stat.size > MAX_SINGLE_READ_SIZE)
return readPipeline(cpath, stat.size, sri, new Pipeline()).concat()
}
return readFile(cpath, null).then((data) => {
if (!ssri.checkData(data, sri)) {
if (!ssri.checkData(data, sri))
throw integrityError(sri, cpath)
}
return data
})
})
@@ -45,11 +39,11 @@ const readPipeline = (cpath, size, sri, stream) => {
stream.push(
new fsm.ReadStream(cpath, {
size,
readSize: MAX_SINGLE_READ_SIZE
readSize: MAX_SINGLE_READ_SIZE,
}),
ssri.integrityStream({
integrity: sri,
size
size,
})
)
return stream
@@ -57,17 +51,15 @@ const readPipeline = (cpath, size, sri, stream) => {
module.exports.sync = readSync
function readSync (cache, integrity, opts) {
opts = ReadOpts(opts)
function readSync (cache, integrity, opts = {}) {
const { size } = opts
return withContentSriSync(cache, integrity, (cpath, sri) => {
const data = fs.readFileSync(cpath)
if (typeof opts.size === 'number' && opts.size !== data.length) {
throw sizeError(opts.size, data.length)
}
if (typeof size === 'number' && size !== data.length)
throw sizeError(size, data.length)
if (ssri.checkData(data, sri)) {
if (ssri.checkData(data, sri))
return data
}
throw integrityError(sri, cpath)
})
@@ -76,17 +68,16 @@ function readSync (cache, integrity, opts) {
module.exports.stream = readStream
module.exports.readStream = readStream
function readStream (cache, integrity, opts) {
opts = ReadOpts(opts)
function readStream (cache, integrity, opts = {}) {
const { size } = opts
const stream = new Pipeline()
withContentSri(cache, integrity, (cpath, sri) => {
// just lstat to ensure it exists
return lstat(cpath).then((stat) => ({ stat, cpath, sri }))
}).then(({ stat, cpath, sri }) => {
if (typeof opts.size === 'number' && opts.size !== stat.size) {
return stream.emit('error', sizeError(opts.size, stat.size))
}
if (typeof size === 'number' && size !== stat.size)
return stream.emit('error', sizeError(size, stat.size))
readPipeline(cpath, stat.size, sri, stream)
}, er => stream.emit('error', er))
@@ -100,15 +91,13 @@ if (fs.copyFile) {
copyFile = util.promisify(fs.copyFile)
}
function copy (cache, integrity, dest, opts) {
opts = ReadOpts(opts)
function copy (cache, integrity, dest) {
return withContentSri(cache, integrity, (cpath, sri) => {
return copyFile(cpath, dest)
})
}
function copySync (cache, integrity, dest, opts) {
opts = ReadOpts(opts)
function copySync (cache, integrity, dest) {
return withContentSriSync(cache, integrity, (cpath, sri) => {
return fs.copyFileSync(cpath, dest)
})
@@ -117,21 +106,21 @@ function copySync (cache, integrity, dest, opts) {
module.exports.hasContent = hasContent
function hasContent (cache, integrity) {
if (!integrity) {
if (!integrity)
return Promise.resolve(false)
}
return withContentSri(cache, integrity, (cpath, sri) => {
return lstat(cpath).then((stat) => ({ size: stat.size, sri, stat }))
}).catch((err) => {
if (err.code === 'ENOENT') {
if (err.code === 'ENOENT')
return false
}
if (err.code === 'EPERM') {
if (process.platform !== 'win32') {
/* istanbul ignore else */
if (process.platform !== 'win32')
throw err
} else {
else
return false
}
}
})
}
@@ -139,23 +128,23 @@ function hasContent (cache, integrity) {
module.exports.hasContent.sync = hasContentSync
function hasContentSync (cache, integrity) {
if (!integrity) {
if (!integrity)
return false
}
return withContentSriSync(cache, integrity, (cpath, sri) => {
try {
const stat = fs.lstatSync(cpath)
return { size: stat.size, sri, stat }
} catch (err) {
if (err.code === 'ENOENT') {
if (err.code === 'ENOENT')
return false
}
if (err.code === 'EPERM') {
if (process.platform !== 'win32') {
/* istanbul ignore else */
if (process.platform !== 'win32')
throw err
} else {
else
return false
}
}
}
})
@@ -173,9 +162,10 @@ function withContentSri (cache, integrity, fn) {
const cpath = contentPath(cache, digests[0])
return fn(cpath, digests[0])
} else {
// Can't use race here because a generic error can happen before a ENOENT error, and can happen before a valid result
// Can't use race here because a generic error can happen before
// a ENOENT error, and can happen before a valid result
return Promise
.all(sri[sri.pickAlgorithm()].map((meta) => {
.all(digests.map((meta) => {
return withContentSri(cache, meta, fn)
.catch((err) => {
if (err.code === 'ENOENT') {
@@ -190,21 +180,16 @@ function withContentSri (cache, integrity, fn) {
.then((results) => {
// Return the first non error if it is found
const result = results.find((r) => !(r instanceof Error))
if (result) {
if (result)
return result
}
// Throw the No matching content found error
const enoentError = results.find((r) => r.code === 'ENOENT')
if (enoentError) {
if (enoentError)
throw enoentError
}
// Throw generic error
const genericError = results.find((r) => r instanceof Error)
if (genericError) {
throw genericError
}
throw results.find((r) => r instanceof Error)
})
}
}
@@ -231,16 +216,14 @@ function withContentSriSync (cache, integrity, fn) {
return fn(cpath, digests[0])
} else {
let lastErr = null
for (const meta of sri[sri.pickAlgorithm()]) {
for (const meta of digests) {
try {
return withContentSriSync(cache, meta, fn)
} catch (err) {
lastErr = err
}
}
if (lastErr) {
throw lastErr
}
throw lastErr
}
}
+4 -7
View File
@@ -10,13 +10,10 @@ module.exports = rm
function rm (cache, integrity) {
return hasContent(cache, integrity).then((content) => {
if (content) {
const sri = content.sri
if (sri) {
return rimraf(contentPath(cache, sri)).then(() => true)
}
} else {
// ~pretty~ sure we can't end up with a content lacking sri, but be safe
if (content && content.sri)
return rimraf(contentPath(cache, content.sri)).then(() => true)
else
return false
}
})
}
+26 -24
View File
@@ -4,7 +4,7 @@ const util = require('util')
const contentPath = require('./path')
const fixOwner = require('../util/fix-owner')
const fs = require('graceful-fs')
const fs = require('fs')
const moveFile = require('../util/move-file')
const Minipass = require('minipass')
const Pipeline = require('minipass-pipeline')
@@ -20,20 +20,17 @@ const writeFile = util.promisify(fs.writeFile)
module.exports = write
function write (cache, data, opts) {
opts = opts || {}
if (opts.algorithms && opts.algorithms.length > 1) {
function write (cache, data, opts = {}) {
const { algorithms, size, integrity } = opts
if (algorithms && algorithms.length > 1)
throw new Error('opts.algorithms only supports a single algorithm for now')
}
if (typeof opts.size === 'number' && data.length !== opts.size) {
return Promise.reject(sizeError(opts.size, data.length))
}
const sri = ssri.fromData(data, {
algorithms: opts.algorithms
})
if (opts.integrity && !ssri.checkData(data, opts.integrity, opts)) {
return Promise.reject(checksumError(opts.integrity, sri))
}
if (typeof size === 'number' && data.length !== size)
return Promise.reject(sizeError(size, data.length))
const sri = ssri.fromData(data, algorithms ? { algorithms } : {})
if (integrity && !ssri.checkData(data, integrity, opts))
return Promise.reject(checksumError(integrity, sri))
return disposer(makeTmp(cache, opts), makeTmpDisposer,
(tmp) => {
@@ -90,8 +87,7 @@ class CacacheWriteStream extends Flush {
}
}
function writeStream (cache, opts) {
opts = opts || {}
function writeStream (cache, opts = {}) {
return new CacacheWriteStream(cache, opts)
}
@@ -115,13 +111,17 @@ function pipeToTmp (inputStream, cache, tmpTarget, opts) {
const hashStream = ssri.integrityStream({
integrity: opts.integrity,
algorithms: opts.algorithms,
size: opts.size
size: opts.size,
})
hashStream.on('integrity', i => {
integrity = i
})
hashStream.on('size', s => {
size = s
})
hashStream.on('integrity', i => { integrity = i })
hashStream.on('size', s => { size = s })
const outStream = new fsm.WriteStream(tmpTarget, {
flags: 'wx'
flags: 'wx',
})
// NB: this can throw if the hashStream has a problem with
@@ -135,21 +135,23 @@ function pipeToTmp (inputStream, cache, tmpTarget, opts) {
return pipeline.promise()
.then(() => ({ integrity, size }))
.catch(er => rimraf(tmpTarget).then(() => { throw er }))
.catch(er => rimraf(tmpTarget).then(() => {
throw er
}))
}
function makeTmp (cache, opts) {
const tmpTarget = uniqueFilename(path.join(cache, 'tmp'), opts.tmpPrefix)
return fixOwner.mkdirfix(cache, path.dirname(tmpTarget)).then(() => ({
target: tmpTarget,
moved: false
moved: false,
}))
}
function makeTmpDisposer (tmp) {
if (tmp.moved) {
if (tmp.moved)
return Promise.resolve()
}
return rimraf(tmp.target)
}
+133 -50
View File
@@ -1,21 +1,27 @@
'use strict'
const util = require('util')
const crypto = require('crypto')
const figgyPudding = require('figgy-pudding')
const fs = require('graceful-fs')
const fs = require('fs')
const Minipass = require('minipass')
const path = require('path')
const ssri = require('ssri')
const uniqueFilename = require('unique-filename')
const { disposer } = require('./util/disposer')
const contentPath = require('./content/path')
const fixOwner = require('./util/fix-owner')
const hashToSegments = require('./util/hash-to-segments')
const indexV = require('../package.json')['cache-version'].index
const moveFile = require('@npmcli/move-file')
const _rimraf = require('rimraf')
const rimraf = util.promisify(_rimraf)
rimraf.sync = _rimraf.sync
const appendFile = util.promisify(fs.appendFile)
const readFile = util.promisify(fs.readFile)
const readdir = util.promisify(fs.readdir)
const writeFile = util.promisify(fs.writeFile)
module.exports.NotFoundError = class NotFoundError extends Error {
constructor (cache, key) {
@@ -26,22 +32,93 @@ module.exports.NotFoundError = class NotFoundError extends Error {
}
}
const IndexOpts = figgyPudding({
metadata: {},
size: {}
})
module.exports.compact = compact
async function compact (cache, key, matchFn, opts = {}) {
const bucket = bucketPath(cache, key)
const entries = await bucketEntries(bucket)
const newEntries = []
// we loop backwards because the bottom-most result is the newest
// since we add new entries with appendFile
for (let i = entries.length - 1; i >= 0; --i) {
const entry = entries[i]
// a null integrity could mean either a delete was appended
// or the user has simply stored an index that does not map
// to any content. we determine if the user wants to keep the
// null integrity based on the validateEntry function passed in options.
// if the integrity is null and no validateEntry is provided, we break
// as we consider the null integrity to be a deletion of everything
// that came before it.
if (entry.integrity === null && !opts.validateEntry)
break
// if this entry is valid, and it is either the first entry or
// the newEntries array doesn't already include an entry that
// matches this one based on the provided matchFn, then we add
// it to the beginning of our list
if ((!opts.validateEntry || opts.validateEntry(entry) === true) &&
(newEntries.length === 0 ||
!newEntries.find((oldEntry) => matchFn(oldEntry, entry))))
newEntries.unshift(entry)
}
const newIndex = '\n' + newEntries.map((entry) => {
const stringified = JSON.stringify(entry)
const hash = hashEntry(stringified)
return `${hash}\t${stringified}`
}).join('\n')
const setup = async () => {
const target = uniqueFilename(path.join(cache, 'tmp'), opts.tmpPrefix)
await fixOwner.mkdirfix(cache, path.dirname(target))
return {
target,
moved: false,
}
}
const teardown = async (tmp) => {
if (!tmp.moved)
return rimraf(tmp.target)
}
const write = async (tmp) => {
await writeFile(tmp.target, newIndex, { flag: 'wx' })
await fixOwner.mkdirfix(cache, path.dirname(bucket))
// we use @npmcli/move-file directly here because we
// want to overwrite the existing file
await moveFile(tmp.target, bucket)
tmp.moved = true
try {
await fixOwner.chownr(cache, bucket)
} catch (err) {
if (err.code !== 'ENOENT')
throw err
}
}
// write the file atomically
await disposer(setup(), teardown, write)
// we reverse the list we generated such that the newest
// entries come first in order to make looping through them easier
// the true passed to formatEntry tells it to keep null
// integrity values, if they made it this far it's because
// validateEntry returned true, and as such we should return it
return newEntries.reverse().map((entry) => formatEntry(cache, entry, true))
}
module.exports.insert = insert
function insert (cache, key, integrity, opts) {
opts = IndexOpts(opts)
function insert (cache, key, integrity, opts = {}) {
const { metadata, size } = opts
const bucket = bucketPath(cache, key)
const entry = {
key,
integrity: integrity && ssri.stringify(integrity),
time: Date.now(),
size: opts.size,
metadata: opts.metadata
size,
metadata,
}
return fixOwner
.mkdirfix(cache, path.dirname(bucket))
@@ -53,14 +130,15 @@ function insert (cache, key, integrity, opts) {
// another while still preserving the string length of the JSON in
// question. So, we just slap the length in there and verify it on read.
//
// Thanks to @isaacs for the whiteboarding session that ended up with this.
// Thanks to @isaacs for the whiteboarding session that ended up with
// this.
return appendFile(bucket, `\n${hashEntry(stringified)}\t${stringified}`)
})
.then(() => fixOwner.chownr(cache, bucket))
.catch((err) => {
if (err.code === 'ENOENT') {
if (err.code === 'ENOENT')
return undefined
}
throw err
// There's a class of race conditions that happen when things get deleted
// during fixOwner, or between the two mkdirfix/chownr calls.
@@ -75,15 +153,15 @@ function insert (cache, key, integrity, opts) {
module.exports.insert.sync = insertSync
function insertSync (cache, key, integrity, opts) {
opts = IndexOpts(opts)
function insertSync (cache, key, integrity, opts = {}) {
const { metadata, size } = opts
const bucket = bucketPath(cache, key)
const entry = {
key,
integrity: integrity && ssri.stringify(integrity),
time: Date.now(),
size: opts.size,
metadata: opts.metadata
size,
metadata,
}
fixOwner.mkdirfix.sync(cache, path.dirname(bucket))
const stringified = JSON.stringify(entry)
@@ -91,9 +169,8 @@ function insertSync (cache, key, integrity, opts) {
try {
fixOwner.chownr.sync(cache, bucket)
} catch (err) {
if (err.code !== 'ENOENT') {
if (err.code !== 'ENOENT')
throw err
}
}
return formatEntry(cache, entry)
}
@@ -105,19 +182,17 @@ function find (cache, key) {
return bucketEntries(bucket)
.then((entries) => {
return entries.reduce((latest, next) => {
if (next && next.key === key) {
if (next && next.key === key)
return formatEntry(cache, next)
} else {
else
return latest
}
}, null)
})
.catch((err) => {
if (err.code === 'ENOENT') {
if (err.code === 'ENOENT')
return null
} else {
else
throw err
}
})
}
@@ -127,31 +202,37 @@ function findSync (cache, key) {
const bucket = bucketPath(cache, key)
try {
return bucketEntriesSync(bucket).reduce((latest, next) => {
if (next && next.key === key) {
if (next && next.key === key)
return formatEntry(cache, next)
} else {
else
return latest
}
}, null)
} catch (err) {
if (err.code === 'ENOENT') {
if (err.code === 'ENOENT')
return null
} else {
else
throw err
}
}
}
module.exports.delete = del
function del (cache, key, opts) {
return insert(cache, key, null, opts)
function del (cache, key, opts = {}) {
if (!opts.removeFully)
return insert(cache, key, null, opts)
const bucket = bucketPath(cache, key)
return rimraf(bucket)
}
module.exports.delete.sync = delSync
function delSync (cache, key, opts) {
return insertSync(cache, key, null, opts)
function delSync (cache, key, opts = {}) {
if (!opts.removeFully)
return insertSync(cache, key, null, opts)
const bucket = bucketPath(cache, key)
return rimraf.sync(bucket)
}
module.exports.lsStream = lsStream
@@ -182,12 +263,12 @@ function lsStream (cache) {
// reduced is a map of key => entry
for (const entry of reduced.values()) {
const formatted = formatEntry(cache, entry)
if (formatted) {
if (formatted)
stream.write(formatted)
}
}
}).catch(err => {
if (err.code === 'ENOENT') { return undefined }
if (err.code === 'ENOENT')
return undefined
throw err
})
})
@@ -215,10 +296,14 @@ function ls (cache) {
)
}
module.exports.bucketEntries = bucketEntries
function bucketEntries (bucket, filter) {
return readFile(bucket, 'utf8').then((data) => _bucketEntries(data, filter))
}
module.exports.bucketEntries.sync = bucketEntriesSync
function bucketEntriesSync (bucket, filter) {
const data = fs.readFileSync(bucket, 'utf8')
return _bucketEntries(data, filter)
@@ -227,9 +312,9 @@ function bucketEntriesSync (bucket, filter) {
function _bucketEntries (data, filter) {
const entries = []
data.split('\n').forEach((entry) => {
if (!entry) {
if (!entry)
return
}
const pieces = entry.split('\t')
if (!pieces[1] || hashEntry(pieces[1]) !== pieces[0]) {
// Hash is no good! Corruption or malice? Doesn't matter!
@@ -243,9 +328,8 @@ function _bucketEntries (data, filter) {
// Entry is corrupted!
return
}
if (obj) {
if (obj)
entries.push(obj)
}
})
return entries
}
@@ -285,26 +369,25 @@ function hash (str, digest) {
.digest('hex')
}
function formatEntry (cache, entry) {
function formatEntry (cache, entry, keepAll) {
// Treat null digests as deletions. They'll shadow any previous entries.
if (!entry.integrity) {
if (!entry.integrity && !keepAll)
return null
}
return {
key: entry.key,
integrity: entry.integrity,
path: contentPath(cache, entry.integrity),
path: entry.integrity ? contentPath(cache, entry.integrity) : undefined,
size: entry.size,
time: entry.time,
metadata: entry.metadata
metadata: entry.metadata,
}
}
function readdirOrEmpty (dir) {
return readdir(dir).catch((err) => {
if (err.code === 'ENOENT' || err.code === 'ENOTDIR') {
if (err.code === 'ENOENT' || err.code === 'ENOTDIR')
return []
}
throw err
})
+5 -6
View File
@@ -8,7 +8,7 @@ const MAX_AGE = 3 * 60 * 1000
const MEMOIZED = new LRU({
max: MAX_SIZE,
maxAge: MAX_AGE,
length: (entry, key) => key.startsWith('key:') ? entry.data.length : entry.length
length: (entry, key) => key.startsWith('key:') ? entry.data.length : entry.length,
})
module.exports.clearMemoized = clearMemoized
@@ -62,13 +62,12 @@ class ObjProxy {
}
function pickMem (opts) {
if (!opts || !opts.memoize) {
if (!opts || !opts.memoize)
return MEMOIZED
} else if (opts.memoize.get && opts.memoize.set) {
else if (opts.memoize.get && opts.memoize.set)
return opts.memoize
} else if (typeof opts.memoize === 'object') {
else if (typeof opts.memoize === 'object')
return new ObjProxy(opts.memoize)
} else {
else
return MEMOIZED
}
}
+2 -2
View File
@@ -8,9 +8,9 @@ function disposer (creatorFn, disposerFn, fn) {
.then(
// disposer resolved, do something with original fn's promise
() => {
if (shouldThrow) {
if (shouldThrow)
throw result
}
return result
},
// Disposer fn failed, crash process
+10 -13
View File
@@ -3,7 +3,7 @@
const util = require('util')
const chownr = util.promisify(require('chownr'))
const mkdirp = util.promisify(require('mkdirp'))
const mkdirp = require('mkdirp')
const inflight = require('promise-inflight')
const inferOwner = require('infer-owner')
@@ -49,9 +49,8 @@ function fixOwner (cache, filepath) {
const { uid, gid } = owner
// No need to override if it's already what we used.
if (self.uid === uid && self.gid === gid) {
if (self.uid === uid && self.gid === gid)
return
}
return inflight('fixOwner: fixing ownership on ' + filepath, () =>
chownr(
@@ -59,9 +58,9 @@ function fixOwner (cache, filepath) {
typeof uid === 'number' ? uid : self.uid,
typeof gid === 'number' ? gid : self.gid
).catch((err) => {
if (err.code === 'ENOENT') {
if (err.code === 'ENOENT')
return null
}
throw err
})
)
@@ -94,9 +93,9 @@ function fixOwnerSync (cache, filepath) {
)
} catch (err) {
// only catch ENOENT, any other error is a problem.
if (err.code === 'ENOENT') {
if (err.code === 'ENOENT')
return null
}
throw err
}
}
@@ -111,14 +110,13 @@ function mkdirfix (cache, p, cb) {
return Promise.resolve(inferOwner(cache)).then(() => {
return mkdirp(p)
.then((made) => {
if (made) {
if (made)
return fixOwner(cache, made).then(() => made)
}
})
.catch((err) => {
if (err.code === 'EEXIST') {
if (err.code === 'EEXIST')
return fixOwner(cache, p).then(() => null)
}
throw err
})
})
@@ -138,8 +136,7 @@ function mkdirfixSync (cache, p) {
if (err.code === 'EEXIST') {
fixOwnerSync(cache, p)
return null
} else {
} else
throw err
}
}
}
+22 -11
View File
@@ -1,16 +1,19 @@
'use strict'
const fs = require('graceful-fs')
const fs = require('fs')
const util = require('util')
const chmod = util.promisify(fs.chmod)
const unlink = util.promisify(fs.unlink)
const stat = util.promisify(fs.stat)
const move = require('move-concurrently')
const move = require('@npmcli/move-file')
const pinflight = require('promise-inflight')
module.exports = moveFile
function moveFile (src, dest) {
const isWindows = global.__CACACHE_TEST_FAKE_WINDOWS__ ||
process.platform === 'win32'
// This isn't quite an fs.rename -- the assumption is that
// if `dest` already exists, and we get certain errors while
// trying to move it, we should just not bother.
@@ -23,22 +26,29 @@ function moveFile (src, dest) {
return new Promise((resolve, reject) => {
fs.link(src, dest, (err) => {
if (err) {
if (err.code === 'EEXIST' || err.code === 'EBUSY') {
if (isWindows && err.code === 'EPERM') {
// XXX This is a really weird way to handle this situation, as it
// results in the src file being deleted even though the dest
// might not exist. Since we pretty much always write files to
// deterministic locations based on content hash, this is likely
// ok (or at worst, just ends in a future cache miss). But it would
// be worth investigating at some time in the future if this is
// really what we want to do here.
return resolve()
} else if (err.code === 'EEXIST' || err.code === 'EBUSY') {
// file already exists, so whatever
} else if (err.code === 'EPERM' && process.platform === 'win32') {
// file handle stayed open even past graceful-fs limits
} else {
return resolve()
} else
return reject(err)
}
}
return resolve()
} else
return resolve()
})
})
.then(() => {
// content should never change for any reason, so make it read-only
return Promise.all([
unlink(src),
process.platform !== 'win32' && chmod(dest, '0444')
!isWindows && chmod(dest, '0444'),
])
})
.catch(() => {
@@ -49,7 +59,8 @@ function moveFile (src, dest) {
throw err
}
// file doesn't already exist! let's try a rename -> copy fallback
return move(src, dest, { Promise, fs })
// only delete if it successfully copies
return move(src, dest)
})
})
})
+12 -19
View File
@@ -1,26 +1,21 @@
'use strict'
const util = require('util')
const fs = require('@npmcli/fs')
const figgyPudding = require('figgy-pudding')
const fixOwner = require('./fix-owner')
const path = require('path')
const rimraf = util.promisify(require('rimraf'))
const uniqueFilename = require('unique-filename')
const { disposer } = require('./disposer')
const TmpOpts = figgyPudding({
tmpPrefix: {}
})
module.exports.mkdir = mktmpdir
function mktmpdir (cache, opts) {
opts = TmpOpts(opts)
const tmpTarget = uniqueFilename(path.join(cache, 'tmp'), opts.tmpPrefix)
return fixOwner.mkdirfix(cache, tmpTarget).then(() => {
return tmpTarget
})
function mktmpdir (cache, opts = {}) {
const { tmpPrefix } = opts
const tmpDir = path.join(cache, 'tmp')
return fs.mkdir(tmpDir, { recursive: true, owner: 'inherit' })
.then(() => {
// do not use path.join(), it drops the trailing / if tmpPrefix is unset
const target = `${tmpDir}${path.sep}${tmpPrefix || ''}`
return fs.mkdtemp(target, { owner: 'inherit' })
})
}
module.exports.withTmp = withTmp
@@ -28,11 +23,9 @@ module.exports.withTmp = withTmp
function withTmp (cache, opts, cb) {
if (!cb) {
cb = opts
opts = null
opts = {}
}
opts = TmpOpts(opts)
return disposer(mktmpdir(cache, opts), rimraf, cb)
return fs.withTempDir(path.join(cache, 'tmp'), cb, opts)
}
module.exports.fix = fixtmpdir
+24 -28
View File
@@ -4,9 +4,8 @@ const util = require('util')
const pMap = require('p-map')
const contentPath = require('./content/path')
const figgyPudding = require('figgy-pudding')
const fixOwner = require('./util/fix-owner')
const fs = require('graceful-fs')
const fs = require('fs')
const fsm = require('fs-minipass')
const glob = util.promisify(require('glob'))
const index = require('./entry-index')
@@ -22,20 +21,16 @@ const truncate = util.promisify(fs.truncate)
const writeFile = util.promisify(fs.writeFile)
const readFile = util.promisify(fs.readFile)
const VerifyOpts = figgyPudding({
concurrency: {
default: 20
},
filter: {},
log: {
default: { silly () {} }
}
const verifyOpts = (opts) => ({
concurrency: 20,
log: { silly () {} },
...opts,
})
module.exports = verify
function verify (cache, opts) {
opts = VerifyOpts(opts)
opts = verifyOpts(opts)
opts.log.silly('verify', 'verifying cache at', cache)
const steps = [
@@ -45,12 +40,12 @@ function verify (cache, opts) {
rebuildIndex,
cleanTmp,
writeVerifile,
markEndTime
markEndTime,
]
return steps
.reduce((promise, step, i) => {
const label = step.name || `step #${i}`
const label = step.name
const start = new Date()
return promise.then((stats) => {
return step(cache, opts).then((s) => {
@@ -59,9 +54,9 @@ function verify (cache, opts) {
stats[k] = s[k]
})
const end = new Date()
if (!stats.runTime) {
if (!stats.runTime)
stats.runTime = {}
}
stats.runTime[label] = end - start
return Promise.resolve(stats)
})
@@ -113,9 +108,9 @@ function garbageCollect (cache, opts) {
const indexStream = index.lsStream(cache)
const liveContent = new Set()
indexStream.on('data', (entry) => {
if (opts.filter && !opts.filter(entry)) {
if (opts.filter && !opts.filter(entry))
return
}
liveContent.add(entry.integrity.toString())
})
return new Promise((resolve, reject) => {
@@ -125,14 +120,14 @@ function garbageCollect (cache, opts) {
return glob(path.join(contentDir, '**'), {
follow: false,
nodir: true,
nosort: true
nosort: true,
}).then((files) => {
return Promise.resolve({
verifiedContent: 0,
reclaimedCount: 0,
reclaimedSize: 0,
badContentCount: 0,
keptSize: 0
keptSize: 0,
}).then((stats) =>
pMap(
files,
@@ -176,14 +171,14 @@ function verifyContent (filepath, sri) {
.then((s) => {
const contentInfo = {
size: s.size,
valid: true
valid: true,
}
return ssri
.checkStream(new fsm.ReadStream(filepath), sri)
.catch((err) => {
if (err.code !== 'EINTEGRITY') {
if (err.code !== 'EINTEGRITY')
throw err
}
return rimraf(filepath).then(() => {
contentInfo.valid = false
})
@@ -191,9 +186,9 @@ function verifyContent (filepath, sri) {
.then(() => contentInfo)
})
.catch((err) => {
if (err.code === 'ENOENT') {
if (err.code === 'ENOENT')
return { size: 0, valid: false }
}
throw err
})
}
@@ -204,18 +199,19 @@ function rebuildIndex (cache, opts) {
const stats = {
missingContent: 0,
rejectedEntries: 0,
totalEntries: 0
totalEntries: 0,
}
const buckets = {}
for (const k in entries) {
/* istanbul ignore else */
if (hasOwnProperty(entries, k)) {
const hashed = index.hashKey(k)
const entry = entries[k]
const excluded = opts.filter && !opts.filter(entry)
excluded && stats.rejectedEntries++
if (buckets[hashed] && !excluded) {
if (buckets[hashed] && !excluded)
buckets[hashed].push(entry)
} else if (buckets[hashed] && excluded) {
else if (buckets[hashed] && excluded) {
// skip
} else if (excluded) {
buckets[hashed] = []
@@ -248,7 +244,7 @@ function rebuildBucket (cache, bucket, stats, opts) {
return index
.insert(cache, entry.key, entry.integrity, {
metadata: entry.metadata,
size: entry.size
size: entry.size,
})
.then(() => {
stats.totalEntries++
+1
View File
@@ -0,0 +1 @@
../mkdirp/bin/cmd.js
+1
View File
@@ -0,0 +1 @@
../rimraf/bin.js
+15
View File
@@ -0,0 +1,15 @@
The ISC License
Copyright (c) Isaac Z. Schlueter and Contributors
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+3
View File
@@ -0,0 +1,3 @@
Like `chown -R`.
Takes the same arguments as `fs.chown()`
+167
View File
@@ -0,0 +1,167 @@
'use strict'
const fs = require('fs')
const path = require('path')
/* istanbul ignore next */
const LCHOWN = fs.lchown ? 'lchown' : 'chown'
/* istanbul ignore next */
const LCHOWNSYNC = fs.lchownSync ? 'lchownSync' : 'chownSync'
/* istanbul ignore next */
const needEISDIRHandled = fs.lchown &&
!process.version.match(/v1[1-9]+\./) &&
!process.version.match(/v10\.[6-9]/)
const lchownSync = (path, uid, gid) => {
try {
return fs[LCHOWNSYNC](path, uid, gid)
} catch (er) {
if (er.code !== 'ENOENT')
throw er
}
}
/* istanbul ignore next */
const chownSync = (path, uid, gid) => {
try {
return fs.chownSync(path, uid, gid)
} catch (er) {
if (er.code !== 'ENOENT')
throw er
}
}
/* istanbul ignore next */
const handleEISDIR =
needEISDIRHandled ? (path, uid, gid, cb) => er => {
// Node prior to v10 had a very questionable implementation of
// fs.lchown, which would always try to call fs.open on a directory
// Fall back to fs.chown in those cases.
if (!er || er.code !== 'EISDIR')
cb(er)
else
fs.chown(path, uid, gid, cb)
}
: (_, __, ___, cb) => cb
/* istanbul ignore next */
const handleEISDirSync =
needEISDIRHandled ? (path, uid, gid) => {
try {
return lchownSync(path, uid, gid)
} catch (er) {
if (er.code !== 'EISDIR')
throw er
chownSync(path, uid, gid)
}
}
: (path, uid, gid) => lchownSync(path, uid, gid)
// fs.readdir could only accept an options object as of node v6
const nodeVersion = process.version
let readdir = (path, options, cb) => fs.readdir(path, options, cb)
let readdirSync = (path, options) => fs.readdirSync(path, options)
/* istanbul ignore next */
if (/^v4\./.test(nodeVersion))
readdir = (path, options, cb) => fs.readdir(path, cb)
const chown = (cpath, uid, gid, cb) => {
fs[LCHOWN](cpath, uid, gid, handleEISDIR(cpath, uid, gid, er => {
// Skip ENOENT error
cb(er && er.code !== 'ENOENT' ? er : null)
}))
}
const chownrKid = (p, child, uid, gid, cb) => {
if (typeof child === 'string')
return fs.lstat(path.resolve(p, child), (er, stats) => {
// Skip ENOENT error
if (er)
return cb(er.code !== 'ENOENT' ? er : null)
stats.name = child
chownrKid(p, stats, uid, gid, cb)
})
if (child.isDirectory()) {
chownr(path.resolve(p, child.name), uid, gid, er => {
if (er)
return cb(er)
const cpath = path.resolve(p, child.name)
chown(cpath, uid, gid, cb)
})
} else {
const cpath = path.resolve(p, child.name)
chown(cpath, uid, gid, cb)
}
}
const chownr = (p, uid, gid, cb) => {
readdir(p, { withFileTypes: true }, (er, children) => {
// any error other than ENOTDIR or ENOTSUP means it's not readable,
// or doesn't exist. give up.
if (er) {
if (er.code === 'ENOENT')
return cb()
else if (er.code !== 'ENOTDIR' && er.code !== 'ENOTSUP')
return cb(er)
}
if (er || !children.length)
return chown(p, uid, gid, cb)
let len = children.length
let errState = null
const then = er => {
if (errState)
return
if (er)
return cb(errState = er)
if (-- len === 0)
return chown(p, uid, gid, cb)
}
children.forEach(child => chownrKid(p, child, uid, gid, then))
})
}
const chownrKidSync = (p, child, uid, gid) => {
if (typeof child === 'string') {
try {
const stats = fs.lstatSync(path.resolve(p, child))
stats.name = child
child = stats
} catch (er) {
if (er.code === 'ENOENT')
return
else
throw er
}
}
if (child.isDirectory())
chownrSync(path.resolve(p, child.name), uid, gid)
handleEISDirSync(path.resolve(p, child.name), uid, gid)
}
const chownrSync = (p, uid, gid) => {
let children
try {
children = readdirSync(p, { withFileTypes: true })
} catch (er) {
if (er.code === 'ENOENT')
return
else if (er.code === 'ENOTDIR' || er.code === 'ENOTSUP')
return handleEISDirSync(p, uid, gid)
else
throw er
}
if (children && children.length)
children.forEach(child => chownrKidSync(p, child, uid, gid))
return handleEISDirSync(p, uid, gid)
}
module.exports = chownr
chownr.sync = chownrSync
+68
View File
@@ -0,0 +1,68 @@
{
"_args": [
[
"chownr@2.0.0",
"/home/node/nuxt"
]
],
"_from": "chownr@2.0.0",
"_id": "chownr@2.0.0",
"_inBundle": false,
"_integrity": "sha512-bIomtDF5KGpdogkLd9VspvFzk9KfpyyGlS8YFVZl7TGPBHL5snIOnxeshwVgPteQ9b4Eydl+pVbIyE1DcvCWgQ==",
"_location": "/cacache/chownr",
"_phantomChildren": {},
"_requested": {
"type": "version",
"registry": true,
"raw": "chownr@2.0.0",
"name": "chownr",
"escapedName": "chownr",
"rawSpec": "2.0.0",
"saveSpec": null,
"fetchSpec": "2.0.0"
},
"_requiredBy": [
"/cacache"
],
"_resolved": "https://registry.npmjs.org/chownr/-/chownr-2.0.0.tgz",
"_spec": "2.0.0",
"_where": "/home/node/nuxt",
"author": {
"name": "Isaac Z. Schlueter",
"email": "i@izs.me",
"url": "http://blog.izs.me/"
},
"bugs": {
"url": "https://github.com/isaacs/chownr/issues"
},
"description": "like `chown -R`",
"devDependencies": {
"mkdirp": "0.3",
"rimraf": "^2.7.1",
"tap": "^14.10.6"
},
"engines": {
"node": ">=10"
},
"files": [
"chownr.js"
],
"homepage": "https://github.com/isaacs/chownr#readme",
"license": "ISC",
"main": "chownr.js",
"name": "chownr",
"repository": {
"type": "git",
"url": "git://github.com/isaacs/chownr.git"
},
"scripts": {
"postversion": "npm publish",
"prepublishOnly": "git push origin --follow-tags",
"preversion": "npm test",
"test": "tap"
},
"tap": {
"check-coverage": true
},
"version": "2.0.0"
}
+15
View File
@@ -0,0 +1,15 @@
The ISC License
Copyright (c) Isaac Z. Schlueter and Contributors
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+166
View File
@@ -0,0 +1,166 @@
# lru cache
A cache object that deletes the least-recently-used items.
[![Build Status](https://travis-ci.org/isaacs/node-lru-cache.svg?branch=master)](https://travis-ci.org/isaacs/node-lru-cache) [![Coverage Status](https://coveralls.io/repos/isaacs/node-lru-cache/badge.svg?service=github)](https://coveralls.io/github/isaacs/node-lru-cache)
## Installation:
```javascript
npm install lru-cache --save
```
## Usage:
```javascript
var LRU = require("lru-cache")
, options = { max: 500
, length: function (n, key) { return n * 2 + key.length }
, dispose: function (key, n) { n.close() }
, maxAge: 1000 * 60 * 60 }
, cache = new LRU(options)
, otherCache = new LRU(50) // sets just the max size
cache.set("key", "value")
cache.get("key") // "value"
// non-string keys ARE fully supported
// but note that it must be THE SAME object, not
// just a JSON-equivalent object.
var someObject = { a: 1 }
cache.set(someObject, 'a value')
// Object keys are not toString()-ed
cache.set('[object Object]', 'a different value')
assert.equal(cache.get(someObject), 'a value')
// A similar object with same keys/values won't work,
// because it's a different object identity
assert.equal(cache.get({ a: 1 }), undefined)
cache.reset() // empty the cache
```
If you put more stuff in it, then items will fall out.
If you try to put an oversized thing in it, then it'll fall out right
away.
## Options
* `max` The maximum size of the cache, checked by applying the length
function to all values in the cache. Not setting this is kind of
silly, since that's the whole purpose of this lib, but it defaults
to `Infinity`. Setting it to a non-number or negative number will
throw a `TypeError`. Setting it to 0 makes it be `Infinity`.
* `maxAge` Maximum age in ms. Items are not pro-actively pruned out
as they age, but if you try to get an item that is too old, it'll
drop it and return undefined instead of giving it to you.
Setting this to a negative value will make everything seem old!
Setting it to a non-number will throw a `TypeError`.
* `length` Function that is used to calculate the length of stored
items. If you're storing strings or buffers, then you probably want
to do something like `function(n, key){return n.length}`. The default is
`function(){return 1}`, which is fine if you want to store `max`
like-sized things. The item is passed as the first argument, and
the key is passed as the second argumnet.
* `dispose` Function that is called on items when they are dropped
from the cache. This can be handy if you want to close file
descriptors or do other cleanup tasks when items are no longer
accessible. Called with `key, value`. It's called *before*
actually removing the item from the internal cache, so if you want
to immediately put it back in, you'll have to do that in a
`nextTick` or `setTimeout` callback or it won't do anything.
* `stale` By default, if you set a `maxAge`, it'll only actually pull
stale items out of the cache when you `get(key)`. (That is, it's
not pre-emptively doing a `setTimeout` or anything.) If you set
`stale:true`, it'll return the stale value before deleting it. If
you don't set this, then it'll return `undefined` when you try to
get a stale entry, as if it had already been deleted.
* `noDisposeOnSet` By default, if you set a `dispose()` method, then
it'll be called whenever a `set()` operation overwrites an existing
key. If you set this option, `dispose()` will only be called when a
key falls out of the cache, not when it is overwritten.
* `updateAgeOnGet` When using time-expiring entries with `maxAge`,
setting this to `true` will make each item's effective time update
to the current time whenever it is retrieved from cache, causing it
to not expire. (It can still fall out of cache based on recency of
use, of course.)
## API
* `set(key, value, maxAge)`
* `get(key) => value`
Both of these will update the "recently used"-ness of the key.
They do what you think. `maxAge` is optional and overrides the
cache `maxAge` option if provided.
If the key is not found, `get()` will return `undefined`.
The key and val can be any value.
* `peek(key)`
Returns the key value (or `undefined` if not found) without
updating the "recently used"-ness of the key.
(If you find yourself using this a lot, you *might* be using the
wrong sort of data structure, but there are some use cases where
it's handy.)
* `del(key)`
Deletes a key out of the cache.
* `reset()`
Clear the cache entirely, throwing away all values.
* `has(key)`
Check if a key is in the cache, without updating the recent-ness
or deleting it for being stale.
* `forEach(function(value,key,cache), [thisp])`
Just like `Array.prototype.forEach`. Iterates over all the keys
in the cache, in order of recent-ness. (Ie, more recently used
items are iterated over first.)
* `rforEach(function(value,key,cache), [thisp])`
The same as `cache.forEach(...)` but items are iterated over in
reverse order. (ie, less recently used items are iterated over
first.)
* `keys()`
Return an array of the keys in the cache.
* `values()`
Return an array of the values in the cache.
* `length`
Return total length of objects in cache taking into account
`length` options function.
* `itemCount`
Return total quantity of objects currently in cache. Note, that
`stale` (see options) items are returned as part of this item
count.
* `dump()`
Return an array of the cache entries ready for serialization and usage
with 'destinationCache.load(arr)`.
* `load(cacheEntriesArray)`
Loads another cache entries array, obtained with `sourceCache.dump()`,
into the cache. The destination cache is reset before loading new entries
* `prune()`
Manually iterates over the entire cache proactively pruning old entries
+334
View File
@@ -0,0 +1,334 @@
'use strict'
// A linked list to keep track of recently-used-ness
const Yallist = require('yallist')
const MAX = Symbol('max')
const LENGTH = Symbol('length')
const LENGTH_CALCULATOR = Symbol('lengthCalculator')
const ALLOW_STALE = Symbol('allowStale')
const MAX_AGE = Symbol('maxAge')
const DISPOSE = Symbol('dispose')
const NO_DISPOSE_ON_SET = Symbol('noDisposeOnSet')
const LRU_LIST = Symbol('lruList')
const CACHE = Symbol('cache')
const UPDATE_AGE_ON_GET = Symbol('updateAgeOnGet')
const naiveLength = () => 1
// lruList is a yallist where the head is the youngest
// item, and the tail is the oldest. the list contains the Hit
// objects as the entries.
// Each Hit object has a reference to its Yallist.Node. This
// never changes.
//
// cache is a Map (or PseudoMap) that matches the keys to
// the Yallist.Node object.
class LRUCache {
constructor (options) {
if (typeof options === 'number')
options = { max: options }
if (!options)
options = {}
if (options.max && (typeof options.max !== 'number' || options.max < 0))
throw new TypeError('max must be a non-negative number')
// Kind of weird to have a default max of Infinity, but oh well.
const max = this[MAX] = options.max || Infinity
const lc = options.length || naiveLength
this[LENGTH_CALCULATOR] = (typeof lc !== 'function') ? naiveLength : lc
this[ALLOW_STALE] = options.stale || false
if (options.maxAge && typeof options.maxAge !== 'number')
throw new TypeError('maxAge must be a number')
this[MAX_AGE] = options.maxAge || 0
this[DISPOSE] = options.dispose
this[NO_DISPOSE_ON_SET] = options.noDisposeOnSet || false
this[UPDATE_AGE_ON_GET] = options.updateAgeOnGet || false
this.reset()
}
// resize the cache when the max changes.
set max (mL) {
if (typeof mL !== 'number' || mL < 0)
throw new TypeError('max must be a non-negative number')
this[MAX] = mL || Infinity
trim(this)
}
get max () {
return this[MAX]
}
set allowStale (allowStale) {
this[ALLOW_STALE] = !!allowStale
}
get allowStale () {
return this[ALLOW_STALE]
}
set maxAge (mA) {
if (typeof mA !== 'number')
throw new TypeError('maxAge must be a non-negative number')
this[MAX_AGE] = mA
trim(this)
}
get maxAge () {
return this[MAX_AGE]
}
// resize the cache when the lengthCalculator changes.
set lengthCalculator (lC) {
if (typeof lC !== 'function')
lC = naiveLength
if (lC !== this[LENGTH_CALCULATOR]) {
this[LENGTH_CALCULATOR] = lC
this[LENGTH] = 0
this[LRU_LIST].forEach(hit => {
hit.length = this[LENGTH_CALCULATOR](hit.value, hit.key)
this[LENGTH] += hit.length
})
}
trim(this)
}
get lengthCalculator () { return this[LENGTH_CALCULATOR] }
get length () { return this[LENGTH] }
get itemCount () { return this[LRU_LIST].length }
rforEach (fn, thisp) {
thisp = thisp || this
for (let walker = this[LRU_LIST].tail; walker !== null;) {
const prev = walker.prev
forEachStep(this, fn, walker, thisp)
walker = prev
}
}
forEach (fn, thisp) {
thisp = thisp || this
for (let walker = this[LRU_LIST].head; walker !== null;) {
const next = walker.next
forEachStep(this, fn, walker, thisp)
walker = next
}
}
keys () {
return this[LRU_LIST].toArray().map(k => k.key)
}
values () {
return this[LRU_LIST].toArray().map(k => k.value)
}
reset () {
if (this[DISPOSE] &&
this[LRU_LIST] &&
this[LRU_LIST].length) {
this[LRU_LIST].forEach(hit => this[DISPOSE](hit.key, hit.value))
}
this[CACHE] = new Map() // hash of items by key
this[LRU_LIST] = new Yallist() // list of items in order of use recency
this[LENGTH] = 0 // length of items in the list
}
dump () {
return this[LRU_LIST].map(hit =>
isStale(this, hit) ? false : {
k: hit.key,
v: hit.value,
e: hit.now + (hit.maxAge || 0)
}).toArray().filter(h => h)
}
dumpLru () {
return this[LRU_LIST]
}
set (key, value, maxAge) {
maxAge = maxAge || this[MAX_AGE]
if (maxAge && typeof maxAge !== 'number')
throw new TypeError('maxAge must be a number')
const now = maxAge ? Date.now() : 0
const len = this[LENGTH_CALCULATOR](value, key)
if (this[CACHE].has(key)) {
if (len > this[MAX]) {
del(this, this[CACHE].get(key))
return false
}
const node = this[CACHE].get(key)
const item = node.value
// dispose of the old one before overwriting
// split out into 2 ifs for better coverage tracking
if (this[DISPOSE]) {
if (!this[NO_DISPOSE_ON_SET])
this[DISPOSE](key, item.value)
}
item.now = now
item.maxAge = maxAge
item.value = value
this[LENGTH] += len - item.length
item.length = len
this.get(key)
trim(this)
return true
}
const hit = new Entry(key, value, len, now, maxAge)
// oversized objects fall out of cache automatically.
if (hit.length > this[MAX]) {
if (this[DISPOSE])
this[DISPOSE](key, value)
return false
}
this[LENGTH] += hit.length
this[LRU_LIST].unshift(hit)
this[CACHE].set(key, this[LRU_LIST].head)
trim(this)
return true
}
has (key) {
if (!this[CACHE].has(key)) return false
const hit = this[CACHE].get(key).value
return !isStale(this, hit)
}
get (key) {
return get(this, key, true)
}
peek (key) {
return get(this, key, false)
}
pop () {
const node = this[LRU_LIST].tail
if (!node)
return null
del(this, node)
return node.value
}
del (key) {
del(this, this[CACHE].get(key))
}
load (arr) {
// reset the cache
this.reset()
const now = Date.now()
// A previous serialized cache has the most recent items first
for (let l = arr.length - 1; l >= 0; l--) {
const hit = arr[l]
const expiresAt = hit.e || 0
if (expiresAt === 0)
// the item was created without expiration in a non aged cache
this.set(hit.k, hit.v)
else {
const maxAge = expiresAt - now
// dont add already expired items
if (maxAge > 0) {
this.set(hit.k, hit.v, maxAge)
}
}
}
}
prune () {
this[CACHE].forEach((value, key) => get(this, key, false))
}
}
const get = (self, key, doUse) => {
const node = self[CACHE].get(key)
if (node) {
const hit = node.value
if (isStale(self, hit)) {
del(self, node)
if (!self[ALLOW_STALE])
return undefined
} else {
if (doUse) {
if (self[UPDATE_AGE_ON_GET])
node.value.now = Date.now()
self[LRU_LIST].unshiftNode(node)
}
}
return hit.value
}
}
const isStale = (self, hit) => {
if (!hit || (!hit.maxAge && !self[MAX_AGE]))
return false
const diff = Date.now() - hit.now
return hit.maxAge ? diff > hit.maxAge
: self[MAX_AGE] && (diff > self[MAX_AGE])
}
const trim = self => {
if (self[LENGTH] > self[MAX]) {
for (let walker = self[LRU_LIST].tail;
self[LENGTH] > self[MAX] && walker !== null;) {
// We know that we're about to delete this one, and also
// what the next least recently used key will be, so just
// go ahead and set it now.
const prev = walker.prev
del(self, walker)
walker = prev
}
}
}
const del = (self, node) => {
if (node) {
const hit = node.value
if (self[DISPOSE])
self[DISPOSE](hit.key, hit.value)
self[LENGTH] -= hit.length
self[CACHE].delete(hit.key)
self[LRU_LIST].removeNode(node)
}
}
class Entry {
constructor (key, value, length, now, maxAge) {
this.key = key
this.value = value
this.length = length
this.now = now
this.maxAge = maxAge || 0
}
}
const forEachStep = (self, fn, node, thisp) => {
let hit = node.value
if (isStale(self, hit)) {
del(self, node)
if (!self[ALLOW_STALE])
hit = undefined
}
if (hit)
fn.call(thisp, hit.value, hit.key, self)
}
module.exports = LRUCache
+72
View File
@@ -0,0 +1,72 @@
{
"_args": [
[
"lru-cache@6.0.0",
"/home/node/nuxt"
]
],
"_from": "lru-cache@6.0.0",
"_id": "lru-cache@6.0.0",
"_inBundle": false,
"_integrity": "sha512-Jo6dJ04CmSjuznwJSS3pUeWmd/H0ffTlkXXgwZi+eq1UCmqQwCh+eLsYOYCwY991i2Fah4h1BEMCx4qThGbsiA==",
"_location": "/cacache/lru-cache",
"_phantomChildren": {},
"_requested": {
"type": "version",
"registry": true,
"raw": "lru-cache@6.0.0",
"name": "lru-cache",
"escapedName": "lru-cache",
"rawSpec": "6.0.0",
"saveSpec": null,
"fetchSpec": "6.0.0"
},
"_requiredBy": [
"/cacache"
],
"_resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-6.0.0.tgz",
"_spec": "6.0.0",
"_where": "/home/node/nuxt",
"author": {
"name": "Isaac Z. Schlueter",
"email": "i@izs.me"
},
"bugs": {
"url": "https://github.com/isaacs/node-lru-cache/issues"
},
"dependencies": {
"yallist": "^4.0.0"
},
"description": "A cache object that deletes the least-recently-used items.",
"devDependencies": {
"benchmark": "^2.1.4",
"tap": "^14.10.7"
},
"engines": {
"node": ">=10"
},
"files": [
"index.js"
],
"homepage": "https://github.com/isaacs/node-lru-cache#readme",
"keywords": [
"mru",
"lru",
"cache"
],
"license": "ISC",
"main": "index.js",
"name": "lru-cache",
"repository": {
"type": "git",
"url": "git://github.com/isaacs/node-lru-cache.git"
},
"scripts": {
"postversion": "npm publish",
"prepublishOnly": "git push origin --follow-tags",
"preversion": "npm test",
"snap": "tap",
"test": "tap"
},
"version": "6.0.0"
}
+15
View File
@@ -0,0 +1,15 @@
# Changers Lorgs!
## 1.0
Full rewrite. Essentially a brand new module.
- Return a promise instead of taking a callback.
- Use native `fs.mkdir(path, { recursive: true })` when available.
- Drop support for outdated Node.js versions. (Technically still works on
Node.js v8, but only 10 and above are officially supported.)
## 0.x
Original and most widely used recursive directory creation implementation
in JavaScript, dating back to 2010.
+21
View File
@@ -0,0 +1,21 @@
Copyright James Halliday (mail@substack.net) and Isaac Z. Schlueter (i@izs.me)
This project is free software released under the MIT license:
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env node
const usage = () => `
usage: mkdirp [DIR1,DIR2..] {OPTIONS}
Create each supplied directory including any necessary parent directories
that don't yet exist.
If the directory already exists, do nothing.
OPTIONS are:
-m<mode> If a directory needs to be created, set the mode as an octal
--mode=<mode> permission string.
-v --version Print the mkdirp version number
-h --help Print this helpful banner
-p --print Print the first directories created for each path provided
--manual Use manual implementation, even if native is available
`
const dirs = []
const opts = {}
let print = false
let dashdash = false
let manual = false
for (const arg of process.argv.slice(2)) {
if (dashdash)
dirs.push(arg)
else if (arg === '--')
dashdash = true
else if (arg === '--manual')
manual = true
else if (/^-h/.test(arg) || /^--help/.test(arg)) {
console.log(usage())
process.exit(0)
} else if (arg === '-v' || arg === '--version') {
console.log(require('../package.json').version)
process.exit(0)
} else if (arg === '-p' || arg === '--print') {
print = true
} else if (/^-m/.test(arg) || /^--mode=/.test(arg)) {
const mode = parseInt(arg.replace(/^(-m|--mode=)/, ''), 8)
if (isNaN(mode)) {
console.error(`invalid mode argument: ${arg}\nMust be an octal number.`)
process.exit(1)
}
opts.mode = mode
} else
dirs.push(arg)
}
const mkdirp = require('../')
const impl = manual ? mkdirp.manual : mkdirp
if (dirs.length === 0)
console.error(usage())
Promise.all(dirs.map(dir => impl(dir, opts)))
.then(made => print ? made.forEach(m => m && console.log(m)) : null)
.catch(er => {
console.error(er.message)
if (er.code)
console.error(' code: ' + er.code)
process.exit(1)
})
+31
View File
@@ -0,0 +1,31 @@
const optsArg = require('./lib/opts-arg.js')
const pathArg = require('./lib/path-arg.js')
const {mkdirpNative, mkdirpNativeSync} = require('./lib/mkdirp-native.js')
const {mkdirpManual, mkdirpManualSync} = require('./lib/mkdirp-manual.js')
const {useNative, useNativeSync} = require('./lib/use-native.js')
const mkdirp = (path, opts) => {
path = pathArg(path)
opts = optsArg(opts)
return useNative(opts)
? mkdirpNative(path, opts)
: mkdirpManual(path, opts)
}
const mkdirpSync = (path, opts) => {
path = pathArg(path)
opts = optsArg(opts)
return useNativeSync(opts)
? mkdirpNativeSync(path, opts)
: mkdirpManualSync(path, opts)
}
mkdirp.sync = mkdirpSync
mkdirp.native = (path, opts) => mkdirpNative(pathArg(path), optsArg(opts))
mkdirp.manual = (path, opts) => mkdirpManual(pathArg(path), optsArg(opts))
mkdirp.nativeSync = (path, opts) => mkdirpNativeSync(pathArg(path), optsArg(opts))
mkdirp.manualSync = (path, opts) => mkdirpManualSync(pathArg(path), optsArg(opts))
module.exports = mkdirp
+29
View File
@@ -0,0 +1,29 @@
const {dirname} = require('path')
const findMade = (opts, parent, path = undefined) => {
// we never want the 'made' return value to be a root directory
if (path === parent)
return Promise.resolve()
return opts.statAsync(parent).then(
st => st.isDirectory() ? path : undefined, // will fail later
er => er.code === 'ENOENT'
? findMade(opts, dirname(parent), parent)
: undefined
)
}
const findMadeSync = (opts, parent, path = undefined) => {
if (path === parent)
return undefined
try {
return opts.statSync(parent).isDirectory() ? path : undefined
} catch (er) {
return er.code === 'ENOENT'
? findMadeSync(opts, dirname(parent), parent)
: undefined
}
}
module.exports = {findMade, findMadeSync}
+64
View File
@@ -0,0 +1,64 @@
const {dirname} = require('path')
const mkdirpManual = (path, opts, made) => {
opts.recursive = false
const parent = dirname(path)
if (parent === path) {
return opts.mkdirAsync(path, opts).catch(er => {
// swallowed by recursive implementation on posix systems
// any other error is a failure
if (er.code !== 'EISDIR')
throw er
})
}
return opts.mkdirAsync(path, opts).then(() => made || path, er => {
if (er.code === 'ENOENT')
return mkdirpManual(parent, opts)
.then(made => mkdirpManual(path, opts, made))
if (er.code !== 'EEXIST' && er.code !== 'EROFS')
throw er
return opts.statAsync(path).then(st => {
if (st.isDirectory())
return made
else
throw er
}, () => { throw er })
})
}
const mkdirpManualSync = (path, opts, made) => {
const parent = dirname(path)
opts.recursive = false
if (parent === path) {
try {
return opts.mkdirSync(path, opts)
} catch (er) {
// swallowed by recursive implementation on posix systems
// any other error is a failure
if (er.code !== 'EISDIR')
throw er
else
return
}
}
try {
opts.mkdirSync(path, opts)
return made || path
} catch (er) {
if (er.code === 'ENOENT')
return mkdirpManualSync(path, opts, mkdirpManualSync(parent, opts, made))
if (er.code !== 'EEXIST' && er.code !== 'EROFS')
throw er
try {
if (!opts.statSync(path).isDirectory())
throw er
} catch (_) {
throw er
}
}
}
module.exports = {mkdirpManual, mkdirpManualSync}
+39
View File
@@ -0,0 +1,39 @@
const {dirname} = require('path')
const {findMade, findMadeSync} = require('./find-made.js')
const {mkdirpManual, mkdirpManualSync} = require('./mkdirp-manual.js')
const mkdirpNative = (path, opts) => {
opts.recursive = true
const parent = dirname(path)
if (parent === path)
return opts.mkdirAsync(path, opts)
return findMade(opts, path).then(made =>
opts.mkdirAsync(path, opts).then(() => made)
.catch(er => {
if (er.code === 'ENOENT')
return mkdirpManual(path, opts)
else
throw er
}))
}
const mkdirpNativeSync = (path, opts) => {
opts.recursive = true
const parent = dirname(path)
if (parent === path)
return opts.mkdirSync(path, opts)
const made = findMadeSync(opts, path)
try {
opts.mkdirSync(path, opts)
return made
} catch (er) {
if (er.code === 'ENOENT')
return mkdirpManualSync(path, opts)
else
throw er
}
}
module.exports = {mkdirpNative, mkdirpNativeSync}
+23
View File
@@ -0,0 +1,23 @@
const { promisify } = require('util')
const fs = require('fs')
const optsArg = opts => {
if (!opts)
opts = { mode: 0o777, fs }
else if (typeof opts === 'object')
opts = { mode: 0o777, fs, ...opts }
else if (typeof opts === 'number')
opts = { mode: opts, fs }
else if (typeof opts === 'string')
opts = { mode: parseInt(opts, 8), fs }
else
throw new TypeError('invalid options argument')
opts.mkdir = opts.mkdir || opts.fs.mkdir || fs.mkdir
opts.mkdirAsync = promisify(opts.mkdir)
opts.stat = opts.stat || opts.fs.stat || fs.stat
opts.statAsync = promisify(opts.stat)
opts.statSync = opts.statSync || opts.fs.statSync || fs.statSync
opts.mkdirSync = opts.mkdirSync || opts.fs.mkdirSync || fs.mkdirSync
return opts
}
module.exports = optsArg
+29
View File
@@ -0,0 +1,29 @@
const platform = process.env.__TESTING_MKDIRP_PLATFORM__ || process.platform
const { resolve, parse } = require('path')
const pathArg = path => {
if (/\0/.test(path)) {
// simulate same failure that node raises
throw Object.assign(
new TypeError('path must be a string without null bytes'),
{
path,
code: 'ERR_INVALID_ARG_VALUE',
}
)
}
path = resolve(path)
if (platform === 'win32') {
const badWinChars = /[*|"<>?:]/
const {root} = parse(path)
if (badWinChars.test(path.substr(root.length))) {
throw Object.assign(new Error('Illegal characters in path.'), {
path,
code: 'EINVAL',
})
}
}
return path
}
module.exports = pathArg
+10
View File
@@ -0,0 +1,10 @@
const fs = require('fs')
const version = process.env.__TESTING_MKDIRP_NODE_VERSION__ || process.version
const versArr = version.replace(/^v/, '').split('.')
const hasNative = +versArr[0] > 10 || +versArr[0] === 10 && +versArr[1] >= 12
const useNative = !hasNative ? () => false : opts => opts.mkdir === fs.mkdir
const useNativeSync = !hasNative ? () => false : opts => opts.mkdirSync === fs.mkdirSync
module.exports = {useNative, useNativeSync}
+78
View File
@@ -0,0 +1,78 @@
{
"_args": [
[
"mkdirp@1.0.4",
"/home/node/nuxt"
]
],
"_from": "mkdirp@1.0.4",
"_id": "mkdirp@1.0.4",
"_inBundle": false,
"_integrity": "sha512-vVqVZQyf3WLx2Shd0qJ9xuvqgAyKPLAiqITEtqW0oIUjzo3PePDd6fW9iFz30ef7Ysp/oiWqbhszeGWW2T6Gzw==",
"_location": "/cacache/mkdirp",
"_phantomChildren": {},
"_requested": {
"type": "version",
"registry": true,
"raw": "mkdirp@1.0.4",
"name": "mkdirp",
"escapedName": "mkdirp",
"rawSpec": "1.0.4",
"saveSpec": null,
"fetchSpec": "1.0.4"
},
"_requiredBy": [
"/cacache"
],
"_resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-1.0.4.tgz",
"_spec": "1.0.4",
"_where": "/home/node/nuxt",
"bin": {
"mkdirp": "bin/cmd.js"
},
"bugs": {
"url": "https://github.com/isaacs/node-mkdirp/issues"
},
"description": "Recursively mkdir, like `mkdir -p`",
"devDependencies": {
"require-inject": "^1.4.4",
"tap": "^14.10.7"
},
"engines": {
"node": ">=10"
},
"files": [
"bin",
"lib",
"index.js"
],
"homepage": "https://github.com/isaacs/node-mkdirp#readme",
"keywords": [
"mkdir",
"directory",
"make dir",
"make",
"dir",
"recursive",
"native"
],
"license": "MIT",
"main": "index.js",
"name": "mkdirp",
"repository": {
"type": "git",
"url": "git+https://github.com/isaacs/node-mkdirp.git"
},
"scripts": {
"postpublish": "git push origin --follow-tags",
"postversion": "npm publish",
"preversion": "npm test",
"snap": "tap",
"test": "tap"
},
"tap": {
"check-coverage": true,
"coverage-map": "map.js"
},
"version": "1.0.4"
}
+266
View File
@@ -0,0 +1,266 @@
# mkdirp
Like `mkdir -p`, but in Node.js!
Now with a modern API and no\* bugs!
<small>\* may contain some bugs</small>
# example
## pow.js
```js
const mkdirp = require('mkdirp')
// return value is a Promise resolving to the first directory created
mkdirp('/tmp/foo/bar/baz').then(made =>
console.log(`made directories, starting with ${made}`))
```
Output (where `/tmp/foo` already exists)
```
made directories, starting with /tmp/foo/bar
```
Or, if you don't have time to wait around for promises:
```js
const mkdirp = require('mkdirp')
// return value is the first directory created
const made = mkdirp.sync('/tmp/foo/bar/baz')
console.log(`made directories, starting with ${made}`)
```
And now /tmp/foo/bar/baz exists, huzzah!
# methods
```js
const mkdirp = require('mkdirp')
```
## mkdirp(dir, [opts]) -> Promise<String | undefined>
Create a new directory and any necessary subdirectories at `dir` with octal
permission string `opts.mode`. If `opts` is a string or number, it will be
treated as the `opts.mode`.
If `opts.mode` isn't specified, it defaults to `0o777 &
(~process.umask())`.
Promise resolves to first directory `made` that had to be created, or
`undefined` if everything already exists. Promise rejects if any errors
are encountered. Note that, in the case of promise rejection, some
directories _may_ have been created, as recursive directory creation is not
an atomic operation.
You can optionally pass in an alternate `fs` implementation by passing in
`opts.fs`. Your implementation should have `opts.fs.mkdir(path, opts, cb)`
and `opts.fs.stat(path, cb)`.
You can also override just one or the other of `mkdir` and `stat` by
passing in `opts.stat` or `opts.mkdir`, or providing an `fs` option that
only overrides one of these.
## mkdirp.sync(dir, opts) -> String|null
Synchronously create a new directory and any necessary subdirectories at
`dir` with octal permission string `opts.mode`. If `opts` is a string or
number, it will be treated as the `opts.mode`.
If `opts.mode` isn't specified, it defaults to `0o777 &
(~process.umask())`.
Returns the first directory that had to be created, or undefined if
everything already exists.
You can optionally pass in an alternate `fs` implementation by passing in
`opts.fs`. Your implementation should have `opts.fs.mkdirSync(path, mode)`
and `opts.fs.statSync(path)`.
You can also override just one or the other of `mkdirSync` and `statSync`
by passing in `opts.statSync` or `opts.mkdirSync`, or providing an `fs`
option that only overrides one of these.
## mkdirp.manual, mkdirp.manualSync
Use the manual implementation (not the native one). This is the default
when the native implementation is not available or the stat/mkdir
implementation is overridden.
## mkdirp.native, mkdirp.nativeSync
Use the native implementation (not the manual one). This is the default
when the native implementation is available and stat/mkdir are not
overridden.
# implementation
On Node.js v10.12.0 and above, use the native `fs.mkdir(p,
{recursive:true})` option, unless `fs.mkdir`/`fs.mkdirSync` has been
overridden by an option.
## native implementation
- If the path is a root directory, then pass it to the underlying
implementation and return the result/error. (In this case, it'll either
succeed or fail, but we aren't actually creating any dirs.)
- Walk up the path statting each directory, to find the first path that
will be created, `made`.
- Call `fs.mkdir(path, { recursive: true })` (or `fs.mkdirSync`)
- If error, raise it to the caller.
- Return `made`.
## manual implementation
- Call underlying `fs.mkdir` implementation, with `recursive: false`
- If error:
- If path is a root directory, raise to the caller and do not handle it
- If ENOENT, mkdirp parent dir, store result as `made`
- stat(path)
- If error, raise original `mkdir` error
- If directory, return `made`
- Else, raise original `mkdir` error
- else
- return `undefined` if a root dir, or `made` if set, or `path`
## windows vs unix caveat
On Windows file systems, attempts to create a root directory (ie, a drive
letter or root UNC path) will fail. If the root directory exists, then it
will fail with `EPERM`. If the root directory does not exist, then it will
fail with `ENOENT`.
On posix file systems, attempts to create a root directory (in recursive
mode) will succeed silently, as it is treated like just another directory
that already exists. (In non-recursive mode, of course, it fails with
`EEXIST`.)
In order to preserve this system-specific behavior (and because it's not as
if we can create the parent of a root directory anyway), attempts to create
a root directory are passed directly to the `fs` implementation, and any
errors encountered are not handled.
## native error caveat
The native implementation (as of at least Node.js v13.4.0) does not provide
appropriate errors in some cases (see
[nodejs/node#31481](https://github.com/nodejs/node/issues/31481) and
[nodejs/node#28015](https://github.com/nodejs/node/issues/28015)).
In order to work around this issue, the native implementation will fall
back to the manual implementation if an `ENOENT` error is encountered.
# choosing a recursive mkdir implementation
There are a few to choose from! Use the one that suits your needs best :D
## use `fs.mkdir(path, {recursive: true}, cb)` if:
- You wish to optimize performance even at the expense of other factors.
- You don't need to know the first dir created.
- You are ok with getting `ENOENT` as the error when some other problem is
the actual cause.
- You can limit your platforms to Node.js v10.12 and above.
- You're ok with using callbacks instead of promises.
- You don't need/want a CLI.
- You don't need to override the `fs` methods in use.
## use this module (mkdirp 1.x) if:
- You need to know the first directory that was created.
- You wish to use the native implementation if available, but fall back
when it's not.
- You prefer promise-returning APIs to callback-taking APIs.
- You want more useful error messages than the native recursive mkdir
provides (at least as of Node.js v13.4), and are ok with re-trying on
`ENOENT` to achieve this.
- You need (or at least, are ok with) a CLI.
- You need to override the `fs` methods in use.
## use [`make-dir`](http://npm.im/make-dir) if:
- You do not need to know the first dir created (and wish to save a few
`stat` calls when using the native implementation for this reason).
- You wish to use the native implementation if available, but fall back
when it's not.
- You prefer promise-returning APIs to callback-taking APIs.
- You are ok with occasionally getting `ENOENT` errors for failures that
are actually related to something other than a missing file system entry.
- You don't need/want a CLI.
- You need to override the `fs` methods in use.
## use mkdirp 0.x if:
- You need to know the first directory that was created.
- You need (or at least, are ok with) a CLI.
- You need to override the `fs` methods in use.
- You're ok with using callbacks instead of promises.
- You are not running on Windows, where the root-level ENOENT errors can
lead to infinite regress.
- You think vinyl just sounds warmer and richer for some weird reason.
- You are supporting truly ancient Node.js versions, before even the advent
of a `Promise` language primitive. (Please don't. You deserve better.)
# cli
This package also ships with a `mkdirp` command.
```
$ mkdirp -h
usage: mkdirp [DIR1,DIR2..] {OPTIONS}
Create each supplied directory including any necessary parent directories
that don't yet exist.
If the directory already exists, do nothing.
OPTIONS are:
-m<mode> If a directory needs to be created, set the mode as an octal
--mode=<mode> permission string.
-v --version Print the mkdirp version number
-h --help Print this helpful banner
-p --print Print the first directories created for each path provided
--manual Use manual implementation, even if native is available
```
# install
With [npm](http://npmjs.org) do:
```
npm install mkdirp
```
to get the library locally, or
```
npm install -g mkdirp
```
to get the command everywhere, or
```
npx mkdirp ...
```
to run the command without installing it globally.
# platform support
This module works on node v8, but only v10 and above are officially
supported, as Node v8 reached its LTS end of life 2020-01-01, which is in
the past, as of this writing.
# license
MIT
+65
View File
@@ -0,0 +1,65 @@
# v3.0
- Add `--preserve-root` option to executable (default true)
- Drop support for Node.js below version 6
# v2.7
- Make `glob` an optional dependency
# 2.6
- Retry on EBUSY on non-windows platforms as well
- Make `rimraf.sync` 10000% more reliable on Windows
# 2.5
- Handle Windows EPERM when lstat-ing read-only dirs
- Add glob option to pass options to glob
# 2.4
- Add EPERM to delay/retry loop
- Add `disableGlob` option
# 2.3
- Make maxBusyTries and emfileWait configurable
- Handle weird SunOS unlink-dir issue
- Glob the CLI arg for better Windows support
# 2.2
- Handle ENOENT properly on Windows
- Allow overriding fs methods
- Treat EPERM as indicative of non-empty dir
- Remove optional graceful-fs dep
- Consistently return null error instead of undefined on success
- win32: Treat ENOTEMPTY the same as EBUSY
- Add `rimraf` binary
# 2.1
- Fix SunOS error code for a non-empty directory
- Try rmdir before readdir
- Treat EISDIR like EPERM
- Remove chmod
- Remove lstat polyfill, node 0.7 is not supported
# 2.0
- Fix myGid call to check process.getgid
- Simplify the EBUSY backoff logic.
- Use fs.lstat in node >= 0.7.9
- Remove gently option
- remove fiber implementation
- Delete files that are marked read-only
# 1.0
- Allow ENOENT in sync method
- Throw when no callback is provided
- Make opts.gently an absolute path
- use 'stat' if 'lstat' is not available
- Consistent error naming, and rethrow non-ENOENT stat errors
- add fiber implementation
+15
View File
@@ -0,0 +1,15 @@
The ISC License
Copyright (c) Isaac Z. Schlueter and Contributors
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+101
View File
@@ -0,0 +1,101 @@
[![Build Status](https://travis-ci.org/isaacs/rimraf.svg?branch=master)](https://travis-ci.org/isaacs/rimraf) [![Dependency Status](https://david-dm.org/isaacs/rimraf.svg)](https://david-dm.org/isaacs/rimraf) [![devDependency Status](https://david-dm.org/isaacs/rimraf/dev-status.svg)](https://david-dm.org/isaacs/rimraf#info=devDependencies)
The [UNIX command](http://en.wikipedia.org/wiki/Rm_(Unix)) `rm -rf` for node.
Install with `npm install rimraf`, or just drop rimraf.js somewhere.
## API
`rimraf(f, [opts], callback)`
The first parameter will be interpreted as a globbing pattern for files. If you
want to disable globbing you can do so with `opts.disableGlob` (defaults to
`false`). This might be handy, for instance, if you have filenames that contain
globbing wildcard characters.
The callback will be called with an error if there is one. Certain
errors are handled for you:
* Windows: `EBUSY` and `ENOTEMPTY` - rimraf will back off a maximum of
`opts.maxBusyTries` times before giving up, adding 100ms of wait
between each attempt. The default `maxBusyTries` is 3.
* `ENOENT` - If the file doesn't exist, rimraf will return
successfully, since your desired outcome is already the case.
* `EMFILE` - Since `readdir` requires opening a file descriptor, it's
possible to hit `EMFILE` if too many file descriptors are in use.
In the sync case, there's nothing to be done for this. But in the
async case, rimraf will gradually back off with timeouts up to
`opts.emfileWait` ms, which defaults to 1000.
## options
* unlink, chmod, stat, lstat, rmdir, readdir,
unlinkSync, chmodSync, statSync, lstatSync, rmdirSync, readdirSync
In order to use a custom file system library, you can override
specific fs functions on the options object.
If any of these functions are present on the options object, then
the supplied function will be used instead of the default fs
method.
Sync methods are only relevant for `rimraf.sync()`, of course.
For example:
```javascript
var myCustomFS = require('some-custom-fs')
rimraf('some-thing', myCustomFS, callback)
```
* maxBusyTries
If an `EBUSY`, `ENOTEMPTY`, or `EPERM` error code is encountered
on Windows systems, then rimraf will retry with a linear backoff
wait of 100ms longer on each try. The default maxBusyTries is 3.
Only relevant for async usage.
* emfileWait
If an `EMFILE` error is encountered, then rimraf will retry
repeatedly with a linear backoff of 1ms longer on each try, until
the timeout counter hits this max. The default limit is 1000.
If you repeatedly encounter `EMFILE` errors, then consider using
[graceful-fs](http://npm.im/graceful-fs) in your program.
Only relevant for async usage.
* glob
Set to `false` to disable [glob](http://npm.im/glob) pattern
matching.
Set to an object to pass options to the glob module. The default
glob options are `{ nosort: true, silent: true }`.
Glob version 6 is used in this module.
Relevant for both sync and async usage.
* disableGlob
Set to any non-falsey value to disable globbing entirely.
(Equivalent to setting `glob: false`.)
## rimraf.sync
It can remove stuff synchronously, too. But that's not so good. Use
the async API. It's better.
## CLI
If installed with `npm install rimraf -g` it can be used as a global
command `rimraf <path> [<path> ...]` which is useful for cross platform support.
## mkdirp
If you need to create a directory recursively, check out
[mkdirp](https://github.com/substack/node-mkdirp).
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env node
const rimraf = require('./')
const path = require('path')
const isRoot = arg => /^(\/|[a-zA-Z]:\\)$/.test(path.resolve(arg))
const filterOutRoot = arg => {
const ok = preserveRoot === false || !isRoot(arg)
if (!ok) {
console.error(`refusing to remove ${arg}`)
console.error('Set --no-preserve-root to allow this')
}
return ok
}
let help = false
let dashdash = false
let noglob = false
let preserveRoot = true
const args = process.argv.slice(2).filter(arg => {
if (dashdash)
return !!arg
else if (arg === '--')
dashdash = true
else if (arg === '--no-glob' || arg === '-G')
noglob = true
else if (arg === '--glob' || arg === '-g')
noglob = false
else if (arg.match(/^(-+|\/)(h(elp)?|\?)$/))
help = true
else if (arg === '--preserve-root')
preserveRoot = true
else if (arg === '--no-preserve-root')
preserveRoot = false
else
return !!arg
}).filter(arg => !preserveRoot || filterOutRoot(arg))
const go = n => {
if (n >= args.length)
return
const options = noglob ? { glob: false } : {}
rimraf(args[n], options, er => {
if (er)
throw er
go(n+1)
})
}
if (help || args.length === 0) {
// If they didn't ask for help, then this is not a "success"
const log = help ? console.log : console.error
log('Usage: rimraf <path> [<path> ...]')
log('')
log(' Deletes all files and folders at "path" recursively.')
log('')
log('Options:')
log('')
log(' -h, --help Display this usage info')
log(' -G, --no-glob Do not expand glob patterns in arguments')
log(' -g, --glob Expand glob patterns in arguments (default)')
log(' --preserve-root Do not remove \'/\' (default)')
log(' --no-preserve-root Do not treat \'/\' specially')
log(' -- Stop parsing flags')
process.exit(help ? 0 : 1)
} else
go(0)
+73
View File
@@ -0,0 +1,73 @@
{
"_args": [
[
"rimraf@3.0.2",
"/home/node/nuxt"
]
],
"_from": "rimraf@3.0.2",
"_id": "rimraf@3.0.2",
"_inBundle": false,
"_integrity": "sha512-JZkJMZkAGFFPP2YqXZXPbMlMBgsxzE8ILs4lMIX/2o0L9UBw9O/Y3o6wFw/i9YLapcUJWwqbi3kdxIPdC62TIA==",
"_location": "/cacache/rimraf",
"_phantomChildren": {},
"_requested": {
"type": "version",
"registry": true,
"raw": "rimraf@3.0.2",
"name": "rimraf",
"escapedName": "rimraf",
"rawSpec": "3.0.2",
"saveSpec": null,
"fetchSpec": "3.0.2"
},
"_requiredBy": [
"/cacache"
],
"_resolved": "https://registry.npmjs.org/rimraf/-/rimraf-3.0.2.tgz",
"_spec": "3.0.2",
"_where": "/home/node/nuxt",
"author": {
"name": "Isaac Z. Schlueter",
"email": "i@izs.me",
"url": "http://blog.izs.me/"
},
"bin": {
"rimraf": "bin.js"
},
"bugs": {
"url": "https://github.com/isaacs/rimraf/issues"
},
"dependencies": {
"glob": "^7.1.3"
},
"description": "A deep deletion module for node (like `rm -rf`)",
"devDependencies": {
"mkdirp": "^0.5.1",
"tap": "^12.1.1"
},
"files": [
"LICENSE",
"README.md",
"bin.js",
"rimraf.js"
],
"funding": {
"url": "https://github.com/sponsors/isaacs"
},
"homepage": "https://github.com/isaacs/rimraf#readme",
"license": "ISC",
"main": "rimraf.js",
"name": "rimraf",
"repository": {
"type": "git",
"url": "git://github.com/isaacs/rimraf.git"
},
"scripts": {
"postpublish": "git push origin --follow-tags",
"postversion": "npm publish",
"preversion": "npm test",
"test": "tap test/*.js"
},
"version": "3.0.2"
}
+360
View File
@@ -0,0 +1,360 @@
const assert = require("assert")
const path = require("path")
const fs = require("fs")
let glob = undefined
try {
glob = require("glob")
} catch (_err) {
// treat glob as optional.
}
const defaultGlobOpts = {
nosort: true,
silent: true
}
// for EMFILE handling
let timeout = 0
const isWindows = (process.platform === "win32")
const defaults = options => {
const methods = [
'unlink',
'chmod',
'stat',
'lstat',
'rmdir',
'readdir'
]
methods.forEach(m => {
options[m] = options[m] || fs[m]
m = m + 'Sync'
options[m] = options[m] || fs[m]
})
options.maxBusyTries = options.maxBusyTries || 3
options.emfileWait = options.emfileWait || 1000
if (options.glob === false) {
options.disableGlob = true
}
if (options.disableGlob !== true && glob === undefined) {
throw Error('glob dependency not found, set `options.disableGlob = true` if intentional')
}
options.disableGlob = options.disableGlob || false
options.glob = options.glob || defaultGlobOpts
}
const rimraf = (p, options, cb) => {
if (typeof options === 'function') {
cb = options
options = {}
}
assert(p, 'rimraf: missing path')
assert.equal(typeof p, 'string', 'rimraf: path should be a string')
assert.equal(typeof cb, 'function', 'rimraf: callback function required')
assert(options, 'rimraf: invalid options argument provided')
assert.equal(typeof options, 'object', 'rimraf: options should be object')
defaults(options)
let busyTries = 0
let errState = null
let n = 0
const next = (er) => {
errState = errState || er
if (--n === 0)
cb(errState)
}
const afterGlob = (er, results) => {
if (er)
return cb(er)
n = results.length
if (n === 0)
return cb()
results.forEach(p => {
const CB = (er) => {
if (er) {
if ((er.code === "EBUSY" || er.code === "ENOTEMPTY" || er.code === "EPERM") &&
busyTries < options.maxBusyTries) {
busyTries ++
// try again, with the same exact callback as this one.
return setTimeout(() => rimraf_(p, options, CB), busyTries * 100)
}
// this one won't happen if graceful-fs is used.
if (er.code === "EMFILE" && timeout < options.emfileWait) {
return setTimeout(() => rimraf_(p, options, CB), timeout ++)
}
// already gone
if (er.code === "ENOENT") er = null
}
timeout = 0
next(er)
}
rimraf_(p, options, CB)
})
}
if (options.disableGlob || !glob.hasMagic(p))
return afterGlob(null, [p])
options.lstat(p, (er, stat) => {
if (!er)
return afterGlob(null, [p])
glob(p, options.glob, afterGlob)
})
}
// Two possible strategies.
// 1. Assume it's a file. unlink it, then do the dir stuff on EPERM or EISDIR
// 2. Assume it's a directory. readdir, then do the file stuff on ENOTDIR
//
// Both result in an extra syscall when you guess wrong. However, there
// are likely far more normal files in the world than directories. This
// is based on the assumption that a the average number of files per
// directory is >= 1.
//
// If anyone ever complains about this, then I guess the strategy could
// be made configurable somehow. But until then, YAGNI.
const rimraf_ = (p, options, cb) => {
assert(p)
assert(options)
assert(typeof cb === 'function')
// sunos lets the root user unlink directories, which is... weird.
// so we have to lstat here and make sure it's not a dir.
options.lstat(p, (er, st) => {
if (er && er.code === "ENOENT")
return cb(null)
// Windows can EPERM on stat. Life is suffering.
if (er && er.code === "EPERM" && isWindows)
fixWinEPERM(p, options, er, cb)
if (st && st.isDirectory())
return rmdir(p, options, er, cb)
options.unlink(p, er => {
if (er) {
if (er.code === "ENOENT")
return cb(null)
if (er.code === "EPERM")
return (isWindows)
? fixWinEPERM(p, options, er, cb)
: rmdir(p, options, er, cb)
if (er.code === "EISDIR")
return rmdir(p, options, er, cb)
}
return cb(er)
})
})
}
const fixWinEPERM = (p, options, er, cb) => {
assert(p)
assert(options)
assert(typeof cb === 'function')
options.chmod(p, 0o666, er2 => {
if (er2)
cb(er2.code === "ENOENT" ? null : er)
else
options.stat(p, (er3, stats) => {
if (er3)
cb(er3.code === "ENOENT" ? null : er)
else if (stats.isDirectory())
rmdir(p, options, er, cb)
else
options.unlink(p, cb)
})
})
}
const fixWinEPERMSync = (p, options, er) => {
assert(p)
assert(options)
try {
options.chmodSync(p, 0o666)
} catch (er2) {
if (er2.code === "ENOENT")
return
else
throw er
}
let stats
try {
stats = options.statSync(p)
} catch (er3) {
if (er3.code === "ENOENT")
return
else
throw er
}
if (stats.isDirectory())
rmdirSync(p, options, er)
else
options.unlinkSync(p)
}
const rmdir = (p, options, originalEr, cb) => {
assert(p)
assert(options)
assert(typeof cb === 'function')
// try to rmdir first, and only readdir on ENOTEMPTY or EEXIST (SunOS)
// if we guessed wrong, and it's not a directory, then
// raise the original error.
options.rmdir(p, er => {
if (er && (er.code === "ENOTEMPTY" || er.code === "EEXIST" || er.code === "EPERM"))
rmkids(p, options, cb)
else if (er && er.code === "ENOTDIR")
cb(originalEr)
else
cb(er)
})
}
const rmkids = (p, options, cb) => {
assert(p)
assert(options)
assert(typeof cb === 'function')
options.readdir(p, (er, files) => {
if (er)
return cb(er)
let n = files.length
if (n === 0)
return options.rmdir(p, cb)
let errState
files.forEach(f => {
rimraf(path.join(p, f), options, er => {
if (errState)
return
if (er)
return cb(errState = er)
if (--n === 0)
options.rmdir(p, cb)
})
})
})
}
// this looks simpler, and is strictly *faster*, but will
// tie up the JavaScript thread and fail on excessively
// deep directory trees.
const rimrafSync = (p, options) => {
options = options || {}
defaults(options)
assert(p, 'rimraf: missing path')
assert.equal(typeof p, 'string', 'rimraf: path should be a string')
assert(options, 'rimraf: missing options')
assert.equal(typeof options, 'object', 'rimraf: options should be object')
let results
if (options.disableGlob || !glob.hasMagic(p)) {
results = [p]
} else {
try {
options.lstatSync(p)
results = [p]
} catch (er) {
results = glob.sync(p, options.glob)
}
}
if (!results.length)
return
for (let i = 0; i < results.length; i++) {
const p = results[i]
let st
try {
st = options.lstatSync(p)
} catch (er) {
if (er.code === "ENOENT")
return
// Windows can EPERM on stat. Life is suffering.
if (er.code === "EPERM" && isWindows)
fixWinEPERMSync(p, options, er)
}
try {
// sunos lets the root user unlink directories, which is... weird.
if (st && st.isDirectory())
rmdirSync(p, options, null)
else
options.unlinkSync(p)
} catch (er) {
if (er.code === "ENOENT")
return
if (er.code === "EPERM")
return isWindows ? fixWinEPERMSync(p, options, er) : rmdirSync(p, options, er)
if (er.code !== "EISDIR")
throw er
rmdirSync(p, options, er)
}
}
}
const rmdirSync = (p, options, originalEr) => {
assert(p)
assert(options)
try {
options.rmdirSync(p)
} catch (er) {
if (er.code === "ENOENT")
return
if (er.code === "ENOTDIR")
throw originalEr
if (er.code === "ENOTEMPTY" || er.code === "EEXIST" || er.code === "EPERM")
rmkidsSync(p, options)
}
}
const rmkidsSync = (p, options) => {
assert(p)
assert(options)
options.readdirSync(p).forEach(f => rimrafSync(path.join(p, f), options))
// We only end up here once we got ENOTEMPTY at least once, and
// at this point, we are guaranteed to have removed all the kids.
// So, we know that it won't be ENOENT or ENOTDIR or anything else.
// try really hard to delete stuff on windows, because it has a
// PROFOUNDLY annoying habit of not closing handles promptly when
// files are deleted, resulting in spurious ENOTEMPTY errors.
const retries = isWindows ? 100 : 1
let i = 0
do {
let threw = true
try {
const ret = options.rmdirSync(p, options)
threw = false
return ret
} finally {
if (++i < retries && threw)
continue
}
} while (true)
}
module.exports = rimraf
rimraf.sync = rimrafSync
+46 -55
View File
@@ -1,36 +1,35 @@
{
"_args": [
[
"cacache@13.0.1",
"/mnt/Foxconn/Digitalent/Deverloper/liff-push_2series"
"cacache@15.3.0",
"/home/node/nuxt"
]
],
"_from": "cacache@13.0.1",
"_id": "cacache@13.0.1",
"_from": "cacache@15.3.0",
"_id": "cacache@15.3.0",
"_inBundle": false,
"_integrity": "sha512-5ZvAxd05HDDU+y9BVvcqYu2LLXmPnQ0hW62h32g4xBTgL/MppR4/04NHfj/ycM2y6lmTnbw6HVi+1eN0Psba6w==",
"_integrity": "sha512-VVdYzXEn+cnbXpFgWs5hTT7OScegHVmLhJIR8Ufqk3iFD6A6j5iSX1KuBTfNEv4tdJWE2PzA6IVFtcLC7fN9wQ==",
"_location": "/cacache",
"_phantomChildren": {},
"_phantomChildren": {
"glob": "7.2.3",
"yallist": "4.0.0"
},
"_requested": {
"type": "version",
"registry": true,
"raw": "cacache@13.0.1",
"raw": "cacache@15.3.0",
"name": "cacache",
"escapedName": "cacache",
"rawSpec": "13.0.1",
"rawSpec": "15.3.0",
"saveSpec": null,
"fetchSpec": "13.0.1"
"fetchSpec": "15.3.0"
},
"_requiredBy": [
"/terser-webpack-plugin"
],
"_resolved": "https://registry.npmjs.org/cacache/-/cacache-13.0.1.tgz",
"_spec": "13.0.1",
"_where": "/mnt/Foxconn/Digitalent/Deverloper/liff-push_2series",
"author": {
"name": "Kat Marchán",
"email": "kzm@sykosomatic.org"
},
"_resolved": "https://registry.npmjs.org/cacache/-/cacache-15.3.0.tgz",
"_spec": "15.3.0",
"_where": "/home/node/nuxt",
"bugs": {
"url": "https://github.com/npm/cacache/issues"
},
@@ -38,56 +37,41 @@
"content": "2",
"index": "5"
},
"contributors": [
{
"name": "Charlotte Spencer",
"email": "charlottelaspencer@gmail.com"
},
{
"name": "Rebecca Turner",
"email": "me@re-becca.org"
}
],
"dependencies": {
"chownr": "^1.1.2",
"figgy-pudding": "^3.5.1",
"@npmcli/fs": "^1.0.0",
"@npmcli/move-file": "^1.0.1",
"chownr": "^2.0.0",
"fs-minipass": "^2.0.0",
"glob": "^7.1.4",
"graceful-fs": "^4.2.2",
"infer-owner": "^1.0.4",
"lru-cache": "^5.1.1",
"minipass": "^3.0.0",
"lru-cache": "^6.0.0",
"minipass": "^3.1.1",
"minipass-collect": "^1.0.2",
"minipass-flush": "^1.0.5",
"minipass-pipeline": "^1.2.2",
"mkdirp": "^0.5.1",
"move-concurrently": "^1.0.1",
"p-map": "^3.0.0",
"mkdirp": "^1.0.3",
"p-map": "^4.0.0",
"promise-inflight": "^1.0.1",
"rimraf": "^2.7.1",
"ssri": "^7.0.0",
"rimraf": "^3.0.2",
"ssri": "^8.0.1",
"tar": "^6.0.2",
"unique-filename": "^1.1.1"
},
"description": "Fast, fault-tolerant, cross-platform, disk-based, data-agnostic, content-addressable cache.",
"devDependencies": {
"@npmcli/lint": "^1.0.1",
"benchmark": "^2.1.4",
"chalk": "^2.4.2",
"cross-env": "^5.2.1",
"chalk": "^4.0.0",
"require-inject": "^1.4.4",
"standard": "^14.3.0",
"standard-version": "^7.0.0",
"tacks": "^1.3.0",
"tap": "^14.6.9",
"weallbehave": "^1.2.0",
"weallcontribute": "^1.0.9"
"tap": "^15.0.9"
},
"engines": {
"node": ">= 8"
"node": ">= 10"
},
"files": [
"*.js",
"lib",
"locales"
"lib"
],
"homepage": "https://github.com/npm/cacache#readme",
"keywords": [
@@ -114,14 +98,21 @@
},
"scripts": {
"benchmarks": "node test/benchmarks",
"postrelease": "npm publish && git push --follow-tags",
"prerelease": "npm t",
"pretest": "standard",
"release": "standard-version -s",
"test": "tap test/*.js",
"test-docker": "docker run -it --rm --name pacotest -v \"$PWD\":/tmp -w /tmp node:latest npm test",
"update-coc": "weallbehave -o . && git add CODE_OF_CONDUCT.md && git commit -m 'docs(coc): updated CODE_OF_CONDUCT.md'",
"update-contrib": "weallcontribute -o . && git add CONTRIBUTING.md && git commit -m 'docs(contributing): updated CONTRIBUTING.md'"
"coverage": "tap",
"lint": "npm run npmclilint -- \"*.*js\" \"lib/**/*.*js\" \"test/**/*.*js\"",
"lintfix": "npm run lint -- --fix",
"npmclilint": "npmcli-lint",
"postsnap": "npm run lintfix --",
"postversion": "npm publish",
"prepublishOnly": "git push origin --follow-tags",
"preversion": "npm test",
"snap": "tap",
"test": "tap",
"test-docker": "docker run -it --rm --name pacotest -v \"$PWD\":/tmp -w /tmp node:latest npm test"
},
"version": "13.0.1"
"tap": {
"100": true,
"test-regex": "test/[^/]*.js"
},
"version": "15.3.0"
}
+20 -31
View File
@@ -1,6 +1,5 @@
'use strict'
const figgyPudding = require('figgy-pudding')
const index = require('./lib/entry-index')
const memo = require('./lib/memoization')
const write = require('./lib/content/write')
@@ -8,33 +7,23 @@ const Flush = require('minipass-flush')
const { PassThrough } = require('minipass-collect')
const Pipeline = require('minipass-pipeline')
const PutOpts = figgyPudding({
algorithms: {
default: ['sha512']
},
integrity: {},
memoize: {},
metadata: {},
pickAlgorithm: {},
size: {},
tmpPrefix: {},
single: {},
sep: {},
error: {},
strict: {}
const putOpts = (opts) => ({
algorithms: ['sha512'],
...opts,
})
module.exports = putData
function putData (cache, key, data, opts) {
opts = PutOpts(opts)
function putData (cache, key, data, opts = {}) {
const { memoize } = opts
opts = putOpts(opts)
return write(cache, data, opts).then((res) => {
return index
.insert(cache, key, res.integrity, opts.concat({ size: res.size }))
.insert(cache, key, res.integrity, { ...opts, size: res.size })
.then((entry) => {
if (opts.memoize) {
if (memoize)
memo.put(cache, entry, data, opts)
}
return res.integrity
})
})
@@ -42,8 +31,9 @@ function putData (cache, key, data, opts) {
module.exports.stream = putStream
function putStream (cache, key, opts) {
opts = PutOpts(opts)
function putStream (cache, key, opts = {}) {
const { memoize } = opts
opts = putOpts(opts)
let integrity
let size
@@ -51,7 +41,7 @@ function putStream (cache, key, opts) {
const pipeline = new Pipeline()
// first item in the pipeline is the memoizer, because we need
// that to end first and get the collected data.
if (opts.memoize) {
if (memoize) {
const memoizer = new PassThrough().on('collect', data => {
memoData = data
})
@@ -75,19 +65,18 @@ function putStream (cache, key, opts) {
pipeline.push(new Flush({
flush () {
return index
.insert(cache, key, integrity, opts.concat({ size }))
.insert(cache, key, integrity, { ...opts, size })
.then((entry) => {
if (opts.memoize && memoData) {
if (memoize && memoData)
memo.put(cache, entry, memoData, opts)
}
if (integrity) {
if (integrity)
pipeline.emit('integrity', integrity)
}
if (size) {
if (size)
pipeline.emit('size', size)
}
})
}
},
}))
return pipeline
Generated Vendored
+2 -2
View File
@@ -11,9 +11,9 @@ const rmContent = require('./lib/content/rm')
module.exports = entry
module.exports.entry = entry
function entry (cache, key) {
function entry (cache, key, opts) {
memo.clearMemoized()
return index.delete(cache, key)
return index.delete(cache, key, opts)
}
module.exports.content = content