New feature to copy a pad without copying entire history. This is useful to perform a low CPU intensive operation while still copying current pad state.
Before, a malicious user could bypass authorization restrictions
imposed by the authorize hook:
* Step 1: Fetch any resource that the malicious user is authorized to
access (e.g., static content).
* Step 2: Use the signed express_sid cookie generated in step 1 to
create a socket.io connection.
* Step 3: Perform the CLIENT_READY handshake for the desired pad.
* Step 4: Profit!
Now the authorization decision made by the authorize hook is
propagated to SecurityManager so that it can approve or reject
socket.io messages as appropriate.
This also sets up future support for per-user read-only and
modify-only (no create) authorization levels.
Authentication plugins almost always want to read and modify
`settings.users`. The settings can already be accessed in a few other
ways, but this is much more convenient.
The authorization logic determines whether the user has already
successfully authenticated by looking to see if `req.session.user`
exists. If an authentication plugin says that it successfully
authenticated the user but it did not create `req.session.user` then
authentication will re-run for every access, and authorization plugins
will be unable to determine whether the user has been authenticated.
Return a 500 internal server error to prevent these problems.
Before it only logged an error like this:
SyntaxError: Unexpected string in JSON at position XYZ
Now it also logs the filename, making it easier to figure out where
the bad data is:
failed to read file /path/to/etherpad-lite/src/locales/en.json: SyntaxError: Unexpected string in JSON at position XYZ
package to reduce http requests: nice-select,
pad_automatic_reconnect, skin_variants, scroll, caretPosition
rename unorm in tar.json so it can be included
Before, anyone who could create a socket.io connection to Etherpad
could read, modify, and create pads at will without authenticating
first.
The `checkAccess` middleware in `webaccess.js` normally handles
authentication and authorization, but it does not run for `/socket.io`
requests. This means that the connection handler in `socketio.js` must
handle authentication and authorization. However, before this change:
* The handler did not require a signed `express_sid` cookie.
* After loading the express-session state, the handler did not check
to see if the user had authenticated.
Now the handler requires a signed `express_sid` cookie, and it ensures
that `socket.request.session.user` is non-null if authentication is
required. (`socket.request.session.user` is non-null if and only if
the user has authenticated.)
* Move session validity check and session author ID fetch to a
separate function. This separate function can be used by hooks,
making it easier for them to properly determine the author ID.
* Rewrite the remainder of checkAccess. Benefits:
- The function is more readable and maintainable now.
- Vulnerability fix: Before, the session IDs in sessionCookie
were not validated when checking settings.requireSession. Now,
sessionCookie must identify a valid session for the
settings.requireSession test to pass.
- Bug fix: Before, checkAccess would sometimes use the author ID
associated with the token even if sessionCookie identified a
valid session. Now it always uses the author ID associated
with the session if available.
There are two different ways an author ID becomes associated with a
user: either bound to a token or bound to a session ID. (The token and
session ID come from the `token` and `sessionID` cookies, or, in the
case of socket.io messages, from the `token` and `sessionID` message
properties.) When `settings.requireSession` is true or the user is
accessing a group pad, the session ID should be used. Otherwise the
token should be used.
Before this change, the `/p/:pad/import` handler was always using the
token, even when `settings.requireSession` was true. This caused the
following error because a different author ID was bound to the token
versus the session ID:
> Unable to import file into ${pad}. Author ${authorID} exists but he
> never contributed to this pad
This bug was reported in issue #4006. PR #4012 worked around the
problem by binding the same author ID to the token as well as the
session ID.
This change does the following:
* Modifies the import handler to use the session ID to obtain the
author ID (when appropriate).
* Expands the documentation for the SecurityManager checkAccess
function.
* Removes the workaround from PR #4012.
* Cleans up the `bin/createUserSession.js` test script.
Not all authentication plugins require the Authorization header, so it
might not be present in subsequent attempts. (In particular, a reverse
proxy might strip it.)
* Improve the comment describing how the access check works.
* Move the `authenticate` logic to where it is used so that people
don't have to keep jumping back and forth to understand how the
access check works.
* Break up the three steps to reduce the number of indentation
levels and improve readability. This should also make it easier to
implement and review planned future changes.
Includes settings
Includes i18n
Includes a nice notification
Disconnects on rate limit
Includes feeding into metrics/stats
Include console warn to server console.
1. Introduce contentcollector.js backend tests
1. Fix issue with OL LI items not being properly numbered after import
1. Fix issue with nested OL LI items being improperly numbered on export
1. Fix issue with new lines not being introduced after lists in on import #3961
1. Sanitize HTML on the way in (import)
1. Fix ExportHTML CSS because it needs to support OL > LI > OL not OL > OL [The latter being the correct format]
1. Fix backend tests.
4177b3f943 moved the font-face declarations from src/static/css/pad.css to two
imported files (src/static/css/pad/fonts.css, src/static/css/pad/toolbar.css)
in a different directory.
This results in the font files being invoked from CSSes residing in different
directories in the minified and un-minified case. URLs in the src attribute are
relative to the stylesheet path [0], and so we have to start requiring clean-css
to rebase them.
Before this change, the non minified casse worked by chance, because there were
a lot of "..", which ended up resolving to the root of the site anyways.
Fixes#3956
[0] https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face/src
Before this change, a client would require two versions of the same assets (with
and without randomVersionString), wasting resources and triggering all sorts of
hard to debug inconsistencies.
This change should have been part of 95fd5ce2a4 and completes it.
After each Eterpad restart, the clients will request a new version of the
static assets, even if they are not modified. This is the price we pay for
knowing that no stale files are going to be served ever again. We could also
have used a salted hash of the Etherpad version, but we chose the simpler way.
For the rationale behind using a random string at each restart, see #3958.
ACHTUNG: this may prevent caching HTTP proxies to work.
Closes#3955.
"token" is a random token representing the author, of the form
t.randomstring_of_lenght_20. The random string is generated by the client. The
cookie is used for every pad in the web UI, and is not used for HTTP API.
This comes from the discussion at https://github.com/ether/etherpad-lite/issues/3563
In this way, if the browser sends a list of preferred languages via
Accept-Language HTTP header, Etherpad will honor that.
Before this change, Etherpad always forced on the user the language from
padOptions.lang in settings.json.
This reverts a feature that was introduced in 295672f598.
If Etherpad is hosted on Windows the frontend test URI needs to be
/tests/frontend/index.html (docs say .../frontend/), otherwise there is this
error: ERR_TOO_MANY_REDIRECTS.
Fixes#3804.
Starting with Etherpad 1.8.3 we decided to use Colibris as default skin for new
installs. Without this change, when starting with no settings.json file,
Etherpad would (wrongly) use "no-skin".
This change should have been part of 70bc71c0c3.
This is a departure from previous versions, which did not limit import/export
requests. Now such requests are ALWAYS rate limited. The default is 10 requests
per IP each 90 seconds, and also applies to old instances upgraded to 1.8.3.
Administrators can tune the parameters via settings.importExportRateLimiting.
Importing to a pad is allowed only if an author has a session estabilished and
has already contributed to that specific pad. This means that as long as the
user is on the pad (via the browser) then import is possible.
Note that an author session is NOT the same as a group session, which is not
required.
This setting does not apply to API requests, only to /p/$PAD$/import
This change of behaviour is introduced in Etherpad 1.8.3, and cannot be
disabled.
From Etherpad 1.8.3 onwards, the maximum allowed size for a single imported
file will always be bounded.
The maximum allowed size can be configured via importMaxFileSize.
The old loadSettings.js was a way of customizing settings upon load, because
the Settings module did not offer this functionality. But it did not work well,
since all the default settings were not loaded.
Let's get rid of loadSettings.js for the bulk of the tests (the "backend"
specs). For the "container" specs, we'll keep it in place until/if we rewrite
Settings.js making it less brittle.
Sometimes, RFC 6265-compliant [0] web servers may send back a cookie whose value
is enclosed in double quotes, such as:
Set-Cookie: sessionCookie="s.37cf5299fbf981e14121fba3a588c02b,s.2b21517bf50729d8130ab85736a11346"; Version=1; Path=/; Domain=localhost; Discard
Where the double quotes at the start and the end of the header value are just
delimiters. This is perfectly legal: Etherpad parsing logic should cope with
that, and remove the quotes early in the request phase.
Somehow, this does not happen, and in such cases the actual value that
sessionCookie ends up having is:
sessionCookie = '"s.37cf5299fbf981e14121fba3a588c02b,s.2b21517bf50729d8130ab85736a11346"'
As quick measure, let's strip the double quotes (when present).
Note that here we are being minimal, limiting ourselves to just removing quotes
at the start and the end of the string.
Fixes#3819.
Also, see #3820.
[0] https://tools.ietf.org/html/rfc6265
This yields better conversion results, but requires the previous change,
otherwise there would have been difficulties in locating the temporary file
name.
In the next commit, we are going to change the conversion method to
"html:XHTML Writer File:UTF8". Without this change, that conversion method name
would end up in the extension of the temporary file that is created as an
intermediate step. In this way, the file extensione will always stay ".html".
No functional changes, hopefully. Only the extension of the temporary file
should change.
This change is meant to ease using LibreOffice as converter. When LibreOffice
converts a file, it adds some classes to the <title> tag.
This is a quick & dirty way of matching the <title> and comment it out
independently on the classes that are set on it.
Clearing the authorship colors of a document with at least two authors, and then
undoing that action caused a disconnect from the pad.
This change disallows undoing clearing authorship colors in order to prevent
the problem from affecting users, and adds the relative test coverage.
This is a change of behaviour, and is documented in the changelog.
Fixes#2802 (sidestepping it).
The previous syntax caused a deprecation warning on Node 10.
However, due to the very old version of log4js Etherpad is currently using,
customError objects are going to be displayed as { inspect: [Function: inspect] }.
This needs to be addressed later, updating log4js.
Fixes#3834.
For some weird reason, these seem to be part of the original swagger
implementation but tests assume they're turned off.
Perhaps a difference between /rest and /api?
- Tests pass ✅
- Added openapi-backend hook
- Generating OpenAPI v3 definitions for each API version
- Definitions served /api/openapi.json /api/{version}/openapi.json
The saved revision "star" button appeared in the timeslider toolbar too.
This change introduces a second flag "page" in toolbar.menu(), which controls
whether the toolbar is being drawn for a pad or in the timeslider page.
Fixes#3767.
Revision b480416375 fixed a bug, but introduced a regression, and the "save
revision" "star" button started to appear both on the left and the right
toolbar.
This change introduces a flag "whichMenu" in toolbar.menu(), that controls
whether the left, the right or the timeslider toolbar is being drawn.
By specification [0], the if-modified-since HTTP header sent by browsers does
not include milliseconds.
Before this patch, let's say a file was generate at time:
t_real-file = 2020-03-22T02:15:53.548Z (note the fractional seconds)
When issuing a conditional request, the browser would truncate the fractional
part, and only request an if-modified-since with this contents:
t_if-modified-since = 2020-03-22T02:15:53.000Z
The minify() function would return HTTP/304 only if
t_if-modified-since >= t_real-file, but this would never be true unless, by
chance, a file was generated at XX.000Z.
This resulted in that file being minified/compressed again and resent to the
client for no reason. After this patch, the server correctly responds with
HTTP/304 without doing any computation, and the browser uses the cached file.
[0] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since
CleanCSS 3.4.19 had a Regex Denial of Service vulnerability and has to be
updated. The major version bump requires the following changes:
1. Disabling rebase is necessary because otherwise the URLs for the web fonts
become wrong;
EXAMPLE 1:
/static/css/src/static/font/fontawesome-etherpad.woff
instead of
/static/font/fontawesome-etherpad.woff
EXAMPLE 2 (this is more surprising):
/p/src/static/font/opendyslexic.otf
instead of
/static/font/opendyslexic.otf
2. CleanCSS.minify() can either receive a string containing the CSS, or an array
of strings. In that case each array element is interpreted as an absolute
local path from which the CSS file is read.
In version 4.x, CleanCSS API was simplified, eliminating the relativeTo
parameter, and thus we cannot use our already loaded "content" argument, but
we have to wrap the absolute path to the CSS in an array and ask the library
to read it by itself.
Fixes#3616.
For some reason authorInfo is sometimes null, and therefore it is not possible
to get colorId from it.
This resulted in the following stack trace:
[2020-03-16 09:27:17.291] [ERROR] console - (node:1746) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'colorId' of null
at <BASEDIR>/src/node/handler/PadMessageHandler.js:1199:37
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async Promise.all (index 0)
at async handleClientReady (<BASEDIR>/src/node/handler/PadMessageHandler.js:1171:5)
[2020-03-16 09:27:17.291] [ERROR] console - (node:1746) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 76)
[2020-03-16 09:27:19.034] [WARN] message - Dropped message, USERINFO_UPDATE Session not ready.[object Object]
Which is due to a bug in Etherpad that we are not going to solve now.
As a workaround, when this happens, let's set the username to "Anonymous" (if
it is not already set), and colorId to the fixed value "#daf0b2". Warning
messages are written in the logs to signal this condition.
This is no definitive solution, but fixes#3612 (via a workaround).
Before this patch, visiting the read-only URL for a random pad would remove
the "Save Revision" (the "star" icon) from all the other RW pads. The only way
to make it appear again was to restart the server.
This change does not fix the underlying bug: after visiting a read only link
the "star" button would still disapper, but it is explictly reinserted via an
ad-hoc condition.
Fixes#3702
By specification, when settings.allowUnknownFileEnds is true and the user tries
to import a file with an unknown extension (this includes no extension),
Etherpad tries to import it as txt.
This broke in Etherpad 1.8.0, that abruptly terminates the processing with an
UnhandledPromiseRejectionWarning.
This patch restores the intended behaviour, and allows to import as text a file
with an unknown extension (on no extension).
In order to catch the UnhandledPromiseRejectionWarning we had to use
fsp_rename(), which is declared earlier in the code and is promised based
instead of fs.rename(), which is callback based.
Fixes#3710.
Before this change, invoking a non existing API method would return an HTTP/200
response with a JSON payload {"code":3,"message":"no such function"}.
This commit changes the HTTP status code to 404, leaving the payload as-is.
Before:
curl --verbose "http://localhost:9001/api/1/notExisting?apikey=ABCDEF"
< HTTP/1.1 200 OK
< X-Powered-By: Express
[...]
{"code":3,"message":"no such function","data":null}
After:
curl --verbose "http://localhost:9001/api/1/notExisting?apikey=ABCDEF"
< HTTP/1.1 404 OK
< X-Powered-By: Express
[...]
{"code":3,"message":"no such function","data":null}
Fixes#3546.
Steps to reproduce (via HTTP API):
1. create a group via createGroup()
2. create a group pad inside that group via createGroupPad()
3. make that pad public calling setPublicStatus(true)
4. access the pad via a clean web browser (with no sessions)
5. UnhandledPromiseRejectionWarning: apierror: sessionID does not exist
This was due to an overlook in 769933786c: "apierror: sessionID does not
exist" may be a legal condition if we are also visiting a public pad. The
function that could throw that error was sessionManager.getSessionInfo(), and
thus it needed to be inside the try...catch block.
Please note that calling getText() on the pad always return the pad contents,
*even for non-public pads*, because the API bypasses the security checks and
directly talks to the DB layer.
Fixes#3600.
The mechanism used for determining if the application is being served over SSL
is wrapped by the "express-session" library for "express_sid", and manual for
the "language" cookie, but it's very similar in both cases.
The "secure" flag is set if one of these is true:
1. we are directly serving Etherpad over SSL using the native nodejs
functionality, via the "ssl" options in settings.json
2. Etherpad is being served in plaintext by nodejs, but we are using a reverse
proxy for terminating the SSL for us;
In this case, the user has to be instructed to properly set trustProxy: true
in settings.json, and the information wheter the application is over SSL or
not will be extracted from the X-Forwarded-Proto HTTP header.
Please note that this will not be compatible with applications being served over
http and https at the same time.
The change on webaccess.js amends 009b61b338, which did not work when the SSL
termination was performed by a reverse proxy.
Reference for automatic "express_sid" configuration:
https://github.com/expressjs/session/blob/v1.17.0/README.md#cookiesecureCloses#3561.
The "io" cookie is created by socket.io, and its purpose is to offer an handle
to perform load balancing with session stickiness when the library falls back to
long polling or below.
In Etherpad's case, if an operator needs to load balance, he can use the
"express_sid" cookie, and thus "io" is of no use.
Moreover, socket.io API does not offer a way of setting the "secure" flag on it,
and thus is a liability.
Let's simply nuke it.
References:
https://socket.io/docs/using-multiple-nodes/#Sticky-load-balancinghttps://github.com/socketio/socket.io/issues/2276#issuecomment-147184662 (not totally true, actually, see above)
