Also add symlinks from the old `bin/` and `tests/` locations to avoid
breaking scripts and other tools.
Motivations:
* Scripts and tests no longer have to do dubious things like:
require('ep_etherpad-lite/node_modules/foo')
to access packages installed as dependencies in
`src/package.json`.
* Plugins can access the backend test helper library in a non-hacky
way:
require('ep_etherpad-lite/tests/backend/common')
* We can delete the top-level `package.json` without breaking our
ability to lint the files in `bin/` and `tests/`.
Deleting the top-level `package.json` has downsides: It will cause
`npm` to print warnings whenever plugins are installed, npm will
no longer be able to enforce a plugin's peer dependency on
ep_etherpad-lite, and npm will keep deleting the
`node_modules/ep_etherpad-lite` symlink that points to `../src`.
But there are significant upsides to deleting the top-level
`package.json`: It will drastically speed up plugin installation
because `npm` doesn't have to recursively walk the dependencies in
`src/package.json`. Also, deleting the top-level `package.json`
avoids npm's horrible dependency hoisting behavior (where it moves
stuff from `src/node_modules/` to the top-level `node_modules/`
directory). Dependency hoisting causes numerous mysterious
problems such as silent failures in `npm outdated` and `npm
update`. Dependency hoisting also breaks plugins that do:
require('ep_etherpad-lite/node_modules/foo')
There are some problems with nyc:
* The coverage numbers aren't useful in our case because most of the
code is executed outside the test process (the test code is mostly
API client logic).
* nyc messes with line numbers, which makes it much harder to debug
problems.
* We're seeing frequent SIGABRT crashes while nyc is printing the
results table. I'm not sure if nyc is the cause of the crashes, or
if it's making a race condition worse, or if the crashes have
nothing to do with nyc, but we don't lose much by removing it so
we might as well see if the crash frequency improves.
Rewrite the `callAll` and `aCallAll` functions to support all
reasonable hook behaviors and to report errors for unreasonable
behaviors (e.g., calling the callback twice).
Now a hook function like the following works as expected when invoked
by `aCallAll`:
```
exports.myHookFn = (hookName, context, cb) => {
cb('some value');
return;
};
```
Rather than reinvent the wheel, use a well-tested library to parse and
write cookies. This should also help prevent XSS vulnerabilities
because the library handles special characters such as semicolon.
Previously Etherpad would not pass the correct client IP address through and this caused the rate limiter to limit users behind reverse proxies. This change allows Etherpad to use a client IP passed from a reverse proxy.
Note to devs: This header can be spoofed and spoofing the header could be used in an attack. To mitigate additional *steps should be taken by Etherpad site admins IE doing rate limiting at proxy.* This only really applies to large scale deployments but it's worth noting.
* `src/node/server.js` can now be run as a script (for normal
operation) or imported as a module (for tests).
* Move shutdown actions to `src/node/server.js` to be close to the
startup actions.
* Put startup and shutdown in functions so that tests can call them.
* Use `await` instead of callbacks.
* Block until the HTTP server is listening to avoid races during
test startup.
* Add a new `shutdown` hook.
* Use the `shutdown` hook to:
* close the HTTP server
* call `end()` on the stats collection to cancel its timers
* call `terminate()` on the Threads.Pool to stop the workers
* Exit with exit code 0 (instead of 1) on SIGTERM.
* Export the HTTP server so that tests can get the HTTP server's
port via `server.address().port` when `settings.port` is 0.
If mocha hangs after running the tests, hit Ctrl-C and wtfnode will
print open files, open sockets, running timers, and running intervals.
Adding an `after` function that closes/stops all of those things will
ensure that mocha exits when it finishes running the tests.
Includes settings
Includes i18n
Includes a nice notification
Disconnects on rate limit
Includes feeding into metrics/stats
Include console warn to server console.
1. Introduce contentcollector.js backend tests
1. Fix issue with OL LI items not being properly numbered after import
1. Fix issue with nested OL LI items being improperly numbered on export
1. Fix issue with new lines not being introduced after lists in on import #3961
1. Sanitize HTML on the way in (import)
1. Fix ExportHTML CSS because it needs to support OL > LI > OL not OL > OL [The latter being the correct format]
1. Fix backend tests.
For the first time in a VERY long time, we now have exactly 0 vulnerabilities
reported by npm audit.
=====
BEFORE:
$ npm audit
=== npm audit security report ===
┌──────────────────────────────────────────────────────────────────────────────┐
│ Manual Review │
│ Some vulnerabilities require your attention to resolve │
│ │
│ Visit https://go.npm.me/audit-guide for additional guidance │
└──────────────────────────────────────────────────────────────────────────────┘
[...]
found 4 low severity vulnerabilities in 13796 scanned packages
4 vulnerabilities require manual review. See the full report for details.
=====
AFTER:
$ npm audit
=== npm audit security report ===
found 0 vulnerabilities
in 13796 scanned packages
This is a departure from previous versions, which did not limit import/export
requests. Now such requests are ALWAYS rate limited. The default is 10 requests
per IP each 90 seconds, and also applies to old instances upgraded to 1.8.3.
Administrators can tune the parameters via settings.importExportRateLimiting.
- Tests pass ✅
- Added openapi-backend hook
- Generating OpenAPI v3 definitions for each API version
- Definitions served /api/openapi.json /api/{version}/openapi.json
No variation in # of security vulnerabilities.
The output of `npm audit` goes from this:
found 7 vulnerabilities (5 low, 2 high) in 13881 scanned packages
7 vulnerabilities require manual review. See the full report for details.
To this:
found 7 vulnerabilities (5 low, 2 high) in 13795 scanned packages
7 vulnerabilities require manual review. See the full report for details.
This is a dev dependency. No impact in production.
After this upgrade the backend tests ("cd src; npm test") still pass.
This fixes 2 reported security vulnerabilities.
The output of `npm audit` goes from this:
found 9 vulnerabilities (7 low, 2 high) in 13707 scanned packages
run `npm audit fix` to fix 1 of them.
1 vulnerability requires semver-major dependency updates.
7 vulnerabilities require manual review. See the full report for details.
To this:
found 7 vulnerabilities (5 low, 2 high) in 13881 scanned packages
7 vulnerabilities require manual review. See the full report for details.
This is a dev dependency. No impact in production.
After this upgrade the backend tests ("cd src; npm test") still pass.
This fixes 1 reported security vulnerability.
The output of `npm audit` goes from this:
found 10 vulnerabilities (8 low, 2 high) in 13390 scanned packages
run `npm audit fix` to fix 2 of them.
1 vulnerability requires semver-major dependency updates.
7 vulnerabilities require manual review. See the full report for details.
To this:
found 9 vulnerabilities (7 low, 2 high) in 13707 scanned packages
run `npm audit fix` to fix 1 of them.
1 vulnerability requires semver-major dependency updates.
7 vulnerabilities require manual review. See the full report for details.
This fixes 107 reported security vulnerabilites.
The output of `npm audit` goes from this:
found 117 vulnerabilities (115 low, 2 high) in 13365 scanned packages
run `npm audit fix` to fix 6 of them.
1 vulnerability requires semver-major dependency updates.
110 vulnerabilities require manual review. See the full report for details.
To this:
found 10 vulnerabilities (8 low, 2 high) in 13390 scanned packages
run `npm audit fix` to fix 2 of them.
1 vulnerability requires semver-major dependency updates.
7 vulnerabilities require manual review. See the full report for details.
The changelog for npm can be read at https://github.com/npm/cli/releases
CleanCSS 3.4.19 had a Regex Denial of Service vulnerability and has to be
updated. The major version bump requires the following changes:
1. Disabling rebase is necessary because otherwise the URLs for the web fonts
become wrong;
EXAMPLE 1:
/static/css/src/static/font/fontawesome-etherpad.woff
instead of
/static/font/fontawesome-etherpad.woff
EXAMPLE 2 (this is more surprising):
/p/src/static/font/opendyslexic.otf
instead of
/static/font/opendyslexic.otf
2. CleanCSS.minify() can either receive a string containing the CSS, or an array
of strings. In that case each array element is interpreted as an absolute
local path from which the CSS file is read.
In version 4.x, CleanCSS API was simplified, eliminating the relativeTo
parameter, and thus we cannot use our already loaded "content" argument, but
we have to wrap the absolute path to the CSS in an array and ask the library
to read it by itself.
Fixes#3616.
Note by muxator:
This commit introduced a copied & modified version of the testing files
loadSettings.js and pad.js.
It's Christmas night, and we want to shipt this feature, so I merged it anyway,
adding a note in both the original and copied files so that hopefully someone
in the distant future is going to merge them back again.
This fixes some security vulnerabilites, among them an arbitrary file overwrite.
The output of `npm audit` goes from this:
found 17 vulnerabilities (15 low, 2 high) in 13344 scanned packages
run `npm audit fix` to fix 6 of them.
1 vulnerability requires semver-major dependency updates.
10 vulnerabilities require manual review. See the full report for details.
To this:
found 5 vulnerabilities (3 low, 2 high) in 13370 scanned packages
1 vulnerability requires semver-major dependency updates.
4 vulnerabilities require manual review. See the full report for details.
Changelog:
- https://github.com/npm/cli/releases
6.13.4 (2019-12-11)
BUGFIXES
320ac9aee npm/bin-links#12 npm/gentle-fs#7 Do not remove global bin/man links inappropriately (@isaacs)
DEPENDENCIES
52fd21061 gentle-fs@2.3.0 (@isaacs)
d06f5c0b0 bin-links@1.1.6 (@isaacs)
6.13.3 (2019-12-09)
DEPENDENCIES
19ce061a2 bin-links@1.1.5 Properly normalize, sanitize, and verify bin entries in package.json.
59c836aae npm-packlist@1.4.7
fb4ecd7d2 pacote@9.5.11
5f33040 #476npm/pacote#22npm/pacote#14 fix: Do not drop perms in git when not root (isaacs, @darcyclarke)
6f229f7 sanitize and normalize package bin field (isaacs)
1743cb339 read-package-json@2.1.1
6.13.2 (2019-12-03)
BUG FIXES
4429645b3 #546 fix docs target typo (@richardlau)
867642942 #142 fix(packageRelativePath): fix 'where' for file deps (@larsgw)
d480f2c17 #527 Revert "windows: Add preliminary WSL support for npm and npx" (@craigloewen-msft)
e4b97962e #504 remove unnecessary package.json read when reading shrinkwrap (@Lighting-Jack)
1c65d26ac #501 fix(fund): open url for string shorthand (@ruyadorno)
ae7afe565 #263 Don't log error message if git tagging is disabled (@woppa684)
4c1b16f6a #182 Warn the user that it is uninstalling npm-install (@Hoidberg)
This upgrade solves the high-severity vulnerabilities regarding
https-proxy-agent that were still present in 8e6bca456f.
The output of `npm audit` goes from this:
found 29 vulnerabilities (3 low, 26 high) in 13338 scanned packages
run `npm audit fix` to fix 4 of them.
1 vulnerability requires semver-major dependency updates.
24 vulnerabilities require manual review. See the full report for details.
To this:
found 5 vulnerabilities (3 low, 2 high) in 13338 scanned packages
1 vulnerability requires semver-major dependency updates.
4 vulnerabilities require manual review. See the full report for details.
Changelog:
- https://github.com/npm/cli/releases
6.13.1 (2019-11-18)
BUG FIXES
938d6124d #472 fix(fund): support funding string shorthand (@ruyadorno)
b49c5535b #471 should not publish tap-snapshot folder (@ruyadorno)
3471d5200 #253 Add preliminary WSL support for npm and npx (@infinnie)
3ef295f23 #486 print quick audit report for human output (@isaacs)
TESTING
dbbf977ac #278 added workflow to trigger and run benchmarks (@mikemimik)
b4f5e3825 #457 feat(docs): adding tests and updating docs to reflect changes in registry teams API. (@nomadtechie)
454c7dd60 #456 fix git configs for git 2.23 and above (@isaacs)
DEPENDENCIES
661d86cd2 make-fetch-happen@5.0.2 (@claudiahdz)
6.13.0 (2019-11-05)
NEW FEATURES
4414b06d9 #273 add fund command (@ruyadorno)
BUG FIXES
e4455409f #281 delete ps1 files on package removal (@NoDocCat)
cd14d4701 #279 update supported node list to remove v6.0, v6.1, v9.0 - v9.2 (@ljharb)
DEPENDENCIES
a37296b20 pacote@9.5.9
d3cb3abe8 read-cmd-shim@1.0.5
TESTING
688cd97be #272 use github actions for CI (@JasonEtco)
9a2d8af84 #240 Clean up some flakiness and inconsistency (@isaacs)
The previous attempt to directly release 1.8.0 had to be hold back, and indeed
1.8.0 was never tagged.
Since 1.8.0 contains many changes, let's do a prerelease instead.
Closes#3660
Without this, on nodejs 10 and 12 (and maybe 8, not tested), Etherpad failed to
start, throwing the following error:
[2019-10-22 19:01:01.439] [ERROR] console - exception thrown: Maximum call stack size exceeded
[2019-10-22 19:01:01.439] [INFO] console - RangeError: Maximum call stack size exceeded
at Function.[Symbol.hasInstance] (<anonymous>)
at ReadStream (/opt/etherpad-lite/src/node_modules/graceful-fs/graceful-fs.js:194:14)
at ReadStream (/opt/etherpad-lite/src/node_modules/graceful-fs/graceful-fs.js:195:28)
at ReadStream (/opt/etherpad-lite/src/node_modules/graceful-fs/graceful-fs.js:195:28)
at ReadStream (/opt/etherpad-lite/src/node_modules/graceful-fs/graceful-fs.js:195:28)
at ReadStream (/opt/etherpad-lite/src/node_modules/graceful-fs/graceful-fs.js:195:28)
at ReadStream (/opt/etherpad-lite/src/node_modules/graceful-fs/graceful-fs.js:195:28)
at ReadStream (/opt/etherpad-lite/src/node_modules/graceful-fs/graceful-fs.js:195:28)
at ReadStream (/opt/etherpad-lite/src/node_modules/graceful-fs/graceful-fs.js:195:28)
at ReadStream (/opt/etherpad-lite/src/node_modules/graceful-fs/graceful-fs.js:195:28)
Fixes#3654.
This upgrade should be backward compatible, but still suffers form major
vulnerabilities in its https-proxy-agent transitive dependency (see
https://www.npmjs.com/advisories/1184).
Changelog:
- https://github.com/npm/cli/releases
6.12.0 (2019-10-08):
Now npm ci runs prepare scripts for git dependencies, and respects the
--no-optional argument. Warnings for engine mismatches are printed again.
Various other fixes and cleanups.
BUG FIXES
890b245dc #252 ci: add dirPacker to options (@claudiahdz)
f3299acd0 #257 npm.community#4792 warn message on engine mismatch
(@ruyadorno)
bbc92fb8f #259 npm.community#10288 Fix figgyPudding error in npm token
(@benblank)
70f54dcb5 #241 doctor: Make OK more consistent (@gemal)
FEATURES
ed993a29c #249 Add CI environment variables to user-agent (@isaacs)
f6b0459a4 #248 Add option to save package-lock without formatting Adds a new
config --format-package-lock, which defaults to true.
(@bl00mber)
DEPENDENCIES
0ca063c5d npm-lifecycle@3.1.4:
fix: filter functions and undefined out of makeEnv (@isaacs)
5df6b0ea2 libcipm@4.0.4:
fix: pack git directories properly (@claudiahdz)
respect no-optional argument (@cruzdanilo)
7e04f728c tar@4.4.12
5c380e5a3 stringify-package@1.0.1 (@isaacs)
62f2ca692 node-gyp@5.0.5 (@isaacs)
0ff0ea47a npm-install-checks@3.0.2 (@isaacs)
f46edae94 hosted-git-info@2.8.5 (@isaacs)
TESTING
44a2b036b #262 fix root-ownership race conditions in meta-test (@isaacs)
6.11.3 (2019-09-03):
Fix npm ci regressions and npm outdated depth.
BUG FIXES
235ed1d28 #239 Don't override user specified depth in outdated. Restores
ability to update packages using --depth as suggested by npm audit. (@G-Rath)
1fafb5151 #242 npm.community#9586 Revert "install: do not descend into
directory deps' child modules" (@isaacs)
cebf542e6 #243 npm.community#9720 ci: pass appropriate configs for file/dir
modes (@isaacs)
DEPENDENCIES
e5fbb7ed1 read-cmd-shim@1.0.4 (@claudiahdz)
23ce65616 npm-pick-manifest@3.0.2 (@claudiahdz)
6.11.2 (2019-08-22):
Fix a recent Windows regression, and two long-standing Windows bugs. Also,
get CI running on Windows, so these things are less likely in the future.
DEPENDENCIES
9778a1b87 cmd-shim@3.0.3: Fix regression where shims fail to preserve exit
code (@isaacs)
bf93e91d8 npm-package-arg@6.1.1: Properly handle git+file: urls on Windows
when a drive letter is included. (@isaacs)
BUGFIXES
6cc4cc66f escape args properly on Windows Bash Despite being bash, Node.js
running on windows git mingw bash still executes child processes
using cmd.exe. As a result, arguments in this environment need to
be escaped in the style of cmd.exe, not bash. (@isaacs)
TESTS
291aba7b8 make tests pass on Windows (@isaacs)
fea3a023a travis: run tests on Windows as well (@isaacs)
6.11.1 (2019-08-20):
Fix a regression for windows command shim syntax.
37db29647 cmd-shim@3.0.2 (@isaacs)
v6.11.0 (2019-08-20):
A few meaty bugfixes, and introducing peerDependenciesMeta.
FEATURES
a12341088 #224 Implements peerDependenciesMeta (@arcanis)
2f3b79bba #234 add new forbidden 403 error code (@claudiahdz)
BUGFIXES
24acc9fc8 and 45772af0d #217 npm.community#8863 npm.community#9327 do not
descend into directory deps' child modules, fix shrinkwrap files
that inappropriately list child nodes of symlink packages (@isaacs
and @salomvary)
50cfe113d #229 fixed typo in semver doc (@gall0ws)
e8fb2a1bd #231 Fix spelling mistakes in CHANGELOG-3.md (@XhmikosR)
769d2e057 npm/uid-number#7 Better error on invalid --user/--group configs.
This addresses the issue when people fail to install binary
packages on Docker and other environments where there is no
'nobody' user. (@isaacs)
8b43c9624 nodejs/node#28987 npm.community#6032 npm.community#6658
npm.community#6069 npm.community#9323 Fix the regression where
random config values in a .npmrc file are not passed to lifecycle
scripts, breaking build processes which rely on them. (@isaacs)
8b85eaa47 save files with inferred ownership rather than relying on SUDO_UID
and SUDO_GID. (@isaacs)
b7f6e5f02 Infer ownership of shrinkwrap files (@isaacs)
54b095d77 #235 Add spec to dist-tag remove function (@theberbie)
DEPENDENCIES
dc8f9e52f pacote@9.5.7: Infer the ownership of all unpacked files in
node_modules, so that we never have user-owned files in root-owned
folders, or root-owned files in user-owned folders. (@isaacs)
bb33940c3 cmd-shim@3.0.0:
9c93ac3 #2 npm#3380 Handle environment variables properly (@basbossink)
2d277f8 #25#36#35 Fix 'no shebang' case by always providing $basedir
in shell script (@igorklopov)
adaf20b #26 Fix $* causing an error when arguments contain parentheses
(@satazor)
49f0c13 #30 Fix paths for MSYS/MINGW bash (@dscho)
51a8af3 #34 Add proper support for PowerShell (@ExE-Boss)
4c37e04 #10 Work around quoted batch file names (@isaacs)
a4e279544 npm-lifecycle@3.1.3 (@isaacs):
fail properly if uid-number raises an error
7086a1809 libcipm@4.0.3 (@isaacs)
8845141f9 read-package-json@2.1.0 (@isaacs)
51c028215 bin-links@1.1.3 (@isaacs)
534a5548c read-cmd-shim@1.0.3 (@isaacs)
3038f2fd5 gentle-fs@2.2.1 (@isaacs)
a609a1648 graceful-fs@4.2.2 (@isaacs)
f0346f754 cacache@12.0.3 (@isaacs)
ca9c615c8 npm-pick-manifest@3.0.0 (@isaacs)
b417affbf pacote@9.5.8 (@isaacs)
TESTS
b6df0913c #228 Proper handing of /usr/bin/node lifecycle-path test (@olivr70)
aaf98e88c npm-registry-mock@1.3.0 (@isaacs)
This upgrade should be backward compatible.
Changelog:
- https://github.com/expressjs/session/blob/master/HISTORY.md
1.17.0 / 2019-10-10
deps: cookie@0.4.0
Add SameSite=None support
deps: safe-buffer@5.2.0
1.16.2 / 2019-06-12
Fix restoring cookie.originalMaxAge when store returns Date
deps: parseurl@~1.3.3
This upgrade should be backward compatible.
Changelogs:
- https://expressjs.com/en/changelog/4x.html
- https://github.com/expressjs/express/blob/master/History.md#4171--2019-05-25
4.17.1 - Release date: 2019-05-25
The 4.17.1 patch release includes one bug fix:
The change to the res.status() API has been reverted due to causing
regressions in existing Express 4 applications.
4.17.0 - Release date: 2019-05-16
The 4.17.0 minor release includes bug fixes and some new features, including:
The express.raw() and express.text() middleware have been added to provide
request body parsing for more raw request payloads. This uses the
expressjs/body-parser module module underneath, so apps that are currently
requiring the module separately can switch to the built-in parsers.
The res.cookie() API now supports the "none" value for the sameSite option.
When the "trust proxy" setting is enabled, the req.hostname now supports
multiple X-Forwarded-For headers in a request.
Starting with this version, Express supports Node.js 10.x and 12.x.
The res.sendFile() API now provides and more immediate and easier to
understand error when a non-string is passed as the path argument.
The res.status() API now provides and more immediate and easier to
understand error when null or undefined is passed as the argument.
Tried to simplify the document, to reduce unneded info, and to use a less
informal language.
For example, the introductory links describing git made sense 10 years ago.
Today they are not needed to understand what Etherpad is.
And mercurial was always better than git, anyways :)
The mailing list and the IRC channel seem pretty dead by now. Let's just
move everything to Github issues, which was the de facto situation anyways.
About the donation links: I am the maintainer, and I do not know the identity of
the owners of the donation links, so it is correct to remove them. The same was
done on the website three months ago:
https://github.com/ether/ether.github.com/commit/d4ef04605da5
This is a non breaking change.
From the changelog (https://github.com/expressjs/session/blob/v1.16.1/HISTORY.md#1161--2019-04-11):
# 1.16.1 / 2019-04-11
- Fix error passing data option to Cookie constructor
- Fix uncaught error from bad session data
# 1.16.0 / 2019-04-10
- Catch invalid cookie.maxAge value earlier
- Deprecate setting cookie.maxAge to a Date object
- Fix issue where resave: false may not save altered sessions
- Remove utils-merge dependency
- Use safe-buffer for improved Buffer API
- Use Set-Cookie as cookie header name for compatibility
- deps: depd@~2.0.0
- Replace internal eval usage with Function constructor
- Use instance methods on process to check for listeners
- perf: remove argument reassignment
- deps: on-headers@~1.0.2
- Fix res.writeHead patch missing return value
This is just a dev dependency, so no real risks, but it's better not to scare
users.
Reported vulnerability before this change:
$ npm audit
=== npm audit security report ===
# Run npm install --save-dev nyc@14.1.0 to resolve 1 vulnerability
SEMVER WARNING: Recommended action is a potentially breaking change
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ High │ Prototype Pollution │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ handlebars │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ nyc [dev] │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ nyc > istanbul-reports > handlebars │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/755 │
└───────────────┴──────────────────────────────────────────────────────────────┘
