Commit graph

1307 commits

Author SHA1 Message Date
Richard Hansen
d4162341e7 webaccess: Always sleep for 1s before returning HTTP 401
Not all authentication plugins require the Authorization header, so it
might not be present in subsequent attempts. (In particular, a reverse
proxy might strip it.)
2020-09-05 22:45:46 +01:00
Richard Hansen
2bf076043f Delete redundant token2author DB save
See:
https://github.com/ether/etherpad-lite/pull/4012#issuecomment-686005563
https://github.com/ether/etherpad-lite/issues/4006
2020-09-05 12:40:16 +01:00
Richard Hansen
e0d6d17bf0 webaccess: Restructure for readability and future changes
* Improve the comment describing how the access check works.
  * Move the `authenticate` logic to where it is used so that people
    don't have to keep jumping back and forth to understand how the
    access check works.
  * Break up the three steps to reduce the number of indentation
    levels and improve readability. This should also make it easier to
    implement and review planned future changes.
2020-09-05 12:37:23 +01:00
Richard Hansen
b044351f0a webaccess: Rename basicAuth to checkAccess
Thanks to hooks, the function can do much more than just basic
authentication.
2020-09-05 12:37:23 +01:00
Richard Hansen
2830aaebf1 webaccess: Use === instead of == for comparison 2020-09-05 12:37:23 +01:00
Richard Hansen
f0e5bff84f webaccess: Simplify object construction 2020-09-05 12:37:23 +01:00
Richard Hansen
0a8569d0ad webaccess: Use const or let instead of var 2020-09-05 12:37:23 +01:00
Richard Hansen
a7be5278a3 webaccess: Use arrow functions instead of function keyword 2020-09-05 12:37:23 +01:00
Richard Hansen
726101d6a8 webaccess: Use single quotes everywhere 2020-09-05 12:37:23 +01:00
Richard Hansen
9011207a37 webaccess: Add semicolons after statements 2020-09-05 12:37:23 +01:00
Richard Hansen
e82a3055e6 webaccess: Whitespace fixes 2020-09-05 12:37:23 +01:00
Richard Hansen
f0b7dc7c53
pluginfw: PadMessageHandler: Pass socket.io Socket object to clientVars hook (#4245)
Also revise the clientVars hook documentation.
2020-09-05 10:51:39 +01:00
John McLear
12c05c20b2
bugfix/import: doc import bugfix (#4235) 2020-08-30 14:11:12 +01:00
webzwo0i
d25cd891ae
remove json2, all supported browsers have JSON now (#4198)
cf. https://developer.mozilla.org/de/docs/Web/JavaScript/Reference/Global_Objects/JSON
2020-07-28 00:52:25 +01:00
John McLear
40014d8230
Rate limit Socket IO communication - WIP (#4036)
Includes settings
    Includes i18n
    Includes a nice notification
    Disconnects on rate limit
    Includes feeding into metrics/stats
    Include console warn to server console.
2020-07-19 22:44:24 +01:00
John McLear
4f5cf2dc63
tests and bugfix: test coverage and fix for 4165 - bad HTML export of list items (#4180)
Also fix for test max value for ratelimiter
2020-07-19 16:31:13 +01:00
webzwo0i
b3680058ff
getChangesetInfo: print error message (#4172) 2020-07-16 23:54:15 +01:00
John McLear
78c97d811c
Script to create session and store token <> author more throughly (dont create ghosts) (#4012) 2020-07-16 10:51:02 +01:00
John McLear
e24ef6969e
Bugfix / tests: Specs list type (#4169)
* remove cruft..  ugh

* housekeeping: close stale issues

* fix issue #3945
2020-07-15 17:37:03 +01:00
John McLear
ab8320d15b
WIP - Test Coverage: Import & Export include LibreOffice Test Coverage (#4163)
Runs on Travis
Will only run locally is ``allowAnyoneToImport`` and ``soffice`` or ``abiword`` is set.
2020-07-14 18:44:53 +01:00
John McLear
b15154cc23
Same site cookie fix - Ready for testing / merge (#3990)
* initial fix for httpprefs

* token

* express_sid fix
2020-07-10 08:43:20 +01:00
b_b
3ea8d571e7
Fix PDF import with libreoffice (#4156)
Fix #4151
2020-07-09 23:50:45 +01:00
b_b
358af5fea1
Fix typo in comment (#4154) 2020-07-09 12:32:54 +01:00
John McLear
a3386e3e55
Dont use jquery in import handler (#4153)
I think jQ3 update broke imports a bit, so this removes jQuery and also ensures only .etherpad does reload of page.
2020-07-08 14:50:48 +01:00
John McLear
8a13bde535
bugfix: #4067 openapi incorrect syntax (#4138)
https://github.com/ether/etherpad-lite/issues/4067#issuecomment-650423412 

Thanks to @tom-chase
2020-06-27 09:47:16 +01:00
John McLear
beccf677a4
bugfix: Fix #4120 where an author might not be populated on restart - if this is the case don't push the author to the array. 2020-06-17 10:54:10 +01:00
John McLear
93829b9e33
pluginfw/performance: Dont cache if max age is 0 (#4098) 2020-06-08 17:21:55 +01:00
John McLear
f1e3aff72f stale code: use terser instead of uglify 2020-06-07 20:09:10 +00:00
John McLear
512a51149d Revert "Merge branch 'develop' of github.com:ether/etherpad-lite into develop"
This reverts commit 6fad0210f8, reversing
changes made to 128f3e15eb.
2020-06-07 18:46:41 +00:00
John McLear
564e8fee07 forcing back to last known stable before parent merge 2020-06-07 18:44:01 +00:00
John McLear
009e1d01f7
staleCode: Async update 2020-06-07 14:56:45 +01:00
John McLear
ea9a9cd883 Revert "Revert "stale: Terser instead of uglify (#4075)""
This reverts commit 423be7f081.
2020-06-07 10:08:11 +00:00
John McLear
aff93d24dd
export: txt import include correct OL prefix (#4086)
Includes test coverage
Final part of solving https://github.com/ether/etherpad-lite/issues/3447
2020-06-06 14:30:13 +01:00
John McLear
423be7f081 Revert "stale: Terser instead of uglify (#4075)"
This reverts commit 552b6d7adb.
2020-06-05 23:55:14 +00:00
John McLear
552b6d7adb
stale: Terser instead of uglify (#4075) 2020-06-05 23:10:24 +01:00
John McLear
d4fcc23fb3
backups: include the pool every 100 revs stored (#3991) 2020-06-05 22:46:12 +01:00
John McLear
a4bdcc3392
tests/editor/ul/li/ol/import/export: Introduce contentcollector.js tests & various OL/UL/LI related bugfixes
1. Introduce contentcollector.js backend tests
1. Fix issue with OL LI items not being properly numbered after import
1. Fix issue with nested OL LI items being improperly numbered on export
1. Fix issue with new lines not being introduced after lists in on import #3961
1. Sanitize HTML on the way in (import)
1. Fix ExportHTML CSS because it needs to support OL > LI > OL not OL > OL [The latter being the correct format]
1. Fix backend tests.
2020-06-05 20:54:16 +01:00
Chocobozzz
c854cced65
performance: Use worker threads to minify JS/CSS files (#3823) 2020-06-04 14:00:50 +01:00
John McLear
0d51e71578
ux: Better ux for if cookies are disabled or not available. 2020-06-01 20:17:48 +01:00
Richard Hansen
07c73d4f2d
webaccess: Log authentication attempts (#4022)
Addresses issue #4016.
2020-06-01 20:11:57 +01:00
John McLear
addb9b957a
import: Resolve error handling import causes instance crash 2020-06-01 20:09:10 +01:00
John McLear
2c944eba34
minification: use mime lookup for mime types (#3979) 2020-06-01 19:37:54 +01:00
John McLear
bfca481b0b
import: setting for allowing import without author existing 2020-06-01 18:19:06 +01:00
Stefan
cffd04446e
admin: Add etherpad update check
This update check notifies admin on startup (via console) or in /admin UI that they need to update Etherpad.
2020-06-01 17:57:53 +01:00
Chocobozzz
59b1a2d5e0
performance: add template cache for quicker page renders 2020-06-01 15:00:47 +01:00
Daniel Krol
f79b50d311
Remove superfluous semicolon (#4037) 2020-05-21 16:43:15 +01:00
Daniel Krol
61c7bb9699
feat(i18n) Custom i18n strings (#4000)
* Custom i18n strings (and some code formatting)

* Documentation for per-instance l10n overwrites
2020-05-19 13:21:31 +01:00
Richard Hansen
901a3f396e Revert "SessionStore: replace password with PASSWORD_HIDDEN when storing in db"
This reverts commit 53f126082a, which
broke user authentication.

Fixes issue #4016.
Reopens issue #3421.
2020-05-17 13:00:17 +02:00
muxator
37abb21a55 minify: rebase relative urls in imported files.
4177b3f943 moved the font-face declarations from src/static/css/pad.css to two
imported files (src/static/css/pad/fonts.css, src/static/css/pad/toolbar.css)
in a different directory.

This results in the font files being invoked from CSSes residing in different
directories in the minified and un-minified case. URLs in the src attribute are
relative to the stylesheet path [0], and so we have to start requiring clean-css
to rebase them.

Before this change, the non minified casse worked by chance, because there were
a lot of "..", which ended up resolving to the root of the site anyways.

Fixes #3956

[0] https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face/src
2020-05-15 01:29:13 +02:00
muxator
6dd9e9adc8 assets: also use cache busting via query string in files imported from acs.js
Before this change, a client would require two versions of the same assets (with
and without randomVersionString), wasting resources and triggering all sorts of
hard to debug inconsistencies.

This change should have been part of 95fd5ce2a4 and completes it.
2020-05-15 01:29:13 +02:00
Sebastian Castro
95fd5ce2a4 assets: add random string to query string to bust cache on restart and deploys
After each Eterpad restart, the clients will request a new version of the
static assets, even if they are not modified. This is the price we pay for
knowing that no stale files are going to be served ever again. We could also
have used a salted hash of the Etherpad version, but we chose the simpler way.

For the rationale behind using a random string at each restart, see #3958.

ACHTUNG: this may prevent caching HTTP proxies to work.

Closes #3955.
2020-05-13 18:58:26 +02:00
Chocobozzz
6cb78e51b0 pad.js: wait write callback instead of buffer callback 2020-04-26 04:14:19 +02:00
Chocobozzz
53003d4471 pad.js: fix freeze on pad deletion when it has many revisions 2020-04-26 04:14:19 +02:00
muxator
01497aa399 SecurityManager: clarify the role of the token parameter (and cookie)
"token" is a random token representing the author, of the form
t.randomstring_of_lenght_20. The random string is generated by the client. The
cookie is used for every pad in the web UI, and is not used for HTTP API.

This comes from the discussion at https://github.com/ether/etherpad-lite/issues/3563
2020-04-24 02:55:20 +02:00
John McLear
208c7a849c pad.html: UI telling the user that a contribution is required before importing
This commit is an integration to 24ee37a38f.
2020-04-22 21:12:49 +02:00
John McLear
c6cb253f76 ImportHandler: UI for showing maxFileSize error on import
This commit is an integration to f4418149cb.
2020-04-14 10:02:21 +00:00
Luke Williams
7ec3be640b specialpages: do not set a language cookie if the browser did not send one
In this way, if the browser sends a list of preferred languages via
Accept-Language HTTP header, Etherpad will honor that.

Before this change, Etherpad always forced on the user the language from
padOptions.lang in settings.json.

This reverts a feature that was introduced in 295672f598.
2020-04-19 19:56:16 +02:00
muxator
9882362e2e settings: clarify that null defaults are supported, using the syntax "${VAR_NAME}"
Using "${VAR_NAME:null}", instead, would define the literal string "null".
2020-04-21 04:44:56 +02:00
Gabriel Augusto Almeida
da0ea3a29d tests: avoid ERR_TOO_MANY_REDIRECTS on frontend tests under Windows
If Etherpad is hosted on Windows the frontend test URI needs to be
/tests/frontend/index.html (docs say .../frontend/), otherwise there is this
error: ERR_TOO_MANY_REDIRECTS.

Fixes #3804.
2020-04-20 01:48:23 +02:00
muxator
48b0290977 AbsolutePaths: be able to read the git commit version independently on CWD
--HG--
branch : absolute-paths
2020-04-19 04:51:50 +02:00
Sebastian Castro
709e5d2233 colibris: introduce skin variants, in order to customize the rendering
This provide a nice way to change the colors of main containers from settings file. See comment inside settings for how it works
2020-04-19 03:03:44 +02:00
muxator
0b0608d7d4 skins: use "colibris" as default, even when no settings.json is present
Starting with Etherpad 1.8.3 we decided to use Colibris as default skin for new
installs. Without this change, when starting with no settings.json file,
Etherpad would (wrongly) use "no-skin".

This change should have been part of 70bc71c0c3.
2020-04-19 02:58:35 +02:00
Christian Schröder
f0fdb94eb0 PadMessageHandler: fix for scoping error hiding original error
`r` is undefined outside of the for loop, but used in the catch block of the try
statement
2020-04-16 02:58:47 +02:00
John McLear
c9d55c81a3 import/export: always rate limit import and exports
This is a departure from previous versions, which did not limit import/export
requests. Now such requests are ALWAYS rate limited. The default is 10 requests
per IP each 90 seconds, and also applies to old instances upgraded to 1.8.3.

Administrators can tune the parameters via settings.importExportRateLimiting.
2020-04-14 03:36:13 +02:00
John McLear
24ee37a38f import: do not allow importing into a pad from the web UI if the user is not on that pad
Importing to a pad is allowed only if an author has a session estabilished and
has already contributed to that specific pad. This means that as long as the
user is on the pad (via the browser) then import is possible.

Note that an author session is NOT the same as a group session, which is not
required.

This setting does not apply to API requests, only to /p/$PAD$/import

This change of behaviour is introduced in Etherpad 1.8.3, and cannot be
disabled.
2020-04-14 03:36:13 +02:00
John McLear
f4418149cb import: introduce importMaxFileSize setting. Defaults to 50 MB
From Etherpad 1.8.3 onwards, the maximum allowed size for a single imported
file will always be bounded.

The maximum allowed size can be configured via importMaxFileSize.
2020-04-14 03:36:13 +02:00
muxator
d1ad29a3d1 importexport: improved logging
This is in preparation to the next activities about import/export securization.
2020-04-14 03:36:13 +02:00
muxator
44186edbc5 tests: remove loadSettings.js for backend tests.
The old loadSettings.js was a way of customizing settings upon load, because
the Settings module did not offer this functionality. But it did not work well,
since all the default settings were not loaded.

Let's get rid of loadSettings.js for the bulk of the tests (the "backend"
specs). For the "container" specs, we'll keep it in place until/if we rewrite
Settings.js making it less brittle.
2020-04-14 03:36:13 +02:00
muxator
cd28643604 express: document the effect of settings.trustProxy 2020-04-14 01:10:19 +02:00
muxator
684f374ece runtime: require node >= 10.13.0 LTS
At the moment, NodeJS 10.x is the lowest supported LTS version. NodeJS 8.x is no
longer supported upstream.

Implements #3835.
Planned in #3650.
2020-04-09 04:43:37 +02:00
Chocobozzz
963d12e614 PadManager: use a set instead of an array in padlist
Avoid looping on the array, especially useful if you have many pads.

--HG--
branch : padlist-use-set
2020-04-09 03:39:32 +02:00
Chocobozzz
94ff21e25c PadManager: anchor the addPad regex to the start of the string
This improves the performance a bit, and is more adherent to the logic of the
application.

--HG--
branch : padlist-use-set
2020-04-09 03:39:32 +02:00
Marcin Cieślak
df08883a00 SecurityManager: remove double quotes from session cookie content
Sometimes, RFC 6265-compliant [0] web servers may send back a cookie whose value
is enclosed in double quotes, such as:

    Set-Cookie: sessionCookie="s.37cf5299fbf981e14121fba3a588c02b,s.2b21517bf50729d8130ab85736a11346"; Version=1; Path=/; Domain=localhost; Discard

Where the double quotes at the start and the end of the header value are just
delimiters. This is perfectly legal: Etherpad parsing logic should cope with
that, and remove the quotes early in the request phase.

Somehow, this does not happen, and in such cases the actual value that
sessionCookie ends up having is:

    sessionCookie = '"s.37cf5299fbf981e14121fba3a588c02b,s.2b21517bf50729d8130ab85736a11346"'

As quick measure, let's strip the double quotes (when present).
Note that here we are being minimal, limiting ourselves to just removing quotes
at the start and the end of the string.

Fixes #3819.
Also, see #3820.


[0] https://tools.ietf.org/html/rfc6265
2020-04-09 01:14:51 +02:00
John McLear
08b83ae358 LibreOffice: use "html:XHTML Writer File:UTF8" export method
This yields better conversion results, but requires the previous change,
otherwise there would have been difficulties in locating the temporary file
name.
2020-04-08 22:51:25 +02:00
John McLear
b2ccd0a191 LibreOffice: decouple the extension of the temporary file from its type
In the next commit, we are going to change the conversion method to
"html:XHTML Writer File:UTF8". Without this change, that conversion method name
would end up in the extension of the temporary file that is created as an
intermediate step. In this way, the file extensione will always stay ".html".

No functional changes, hopefully. Only the extension of the temporary file
should change.
2020-04-08 22:51:25 +02:00
John McLear
a371deb9d1 ImportHandler: quick & dirty way of being more lax when matching <title>
This change is meant to ease using LibreOffice as converter. When LibreOffice
converts a file, it adds some classes to the <title> tag.
This is a quick & dirty way of matching the <title> and comment it out
independently on the classes that are set on it.
2020-04-08 22:51:25 +02:00
John McLear
babf67175c undomodule: disallow undoing "clear authorship colors"
Clearing the authorship colors of a document with at least two authors, and then
undoing that action caused a disconnect from the pad.
This change disallows undoing clearing authorship colors in order to prevent
the problem from affecting users, and adds the relative test coverage.

This is a change of behaviour, and is documented in the changelog.

Fixes #2802 (sidestepping it).
2020-04-08 15:20:37 +02:00
Paul Tiedtke
79406051fa Settings.js: support newlines in default values when using variable substitution
This allows, among other things, to correctly support the configuration of
defaultPadText in Docker via an environment variable.
2020-04-07 04:32:37 +02:00
John McLear
8987c5d813 dependencies: upgrade uglify-js 2.6.2 -> 3.8.1 and adapt Minify
This was a major update that required code changes.
2020-04-03 00:05:15 +00:00
Viljami Kuosmanen
3edd727a94 customError: rewrite the module using class syntax
The previous syntax caused a deprecation warning on Node 10.
However, due to the very old version of log4js Etherpad is currently using,
customError objects are going to be displayed as { inspect: [Function: inspect] }.

This needs to be addressed later, updating log4js.

Fixes #3834.
2020-04-07 02:03:17 +02:00
muxator
4ef59bbda0 api: in getStats(), directly rewrote activePads as an expression
Instead of creating an empty Set and then mutate it.
2020-04-04 22:03:46 +02:00
Chocobozzz
82b919fc65 api: add getStats() function 2020-04-04 22:03:46 +02:00
John McLear
eb45934788 remove noise 2020-04-03 11:32:14 +01:00
John McLear
4e212d12b2 patch fix for 3825 2020-04-03 11:32:14 +01:00
John McLear
c2ea2b3a6d webaccess: do not resave session
Before this change, the database was spammed with session values.
Modern express-session has this baked in.
See https://www.npmjs.com/package/express-session#resave for docs.
2020-04-03 02:55:33 +02:00
John McLear
1f0058dd6f interesting discovery RE 3612 and 2802 2020-04-03 02:40:59 +02:00
Viljami Kuosmanen
ccf406708e openapi: support standard http error codes
API errors are now handled at the end of the request heap by
throwing exceptions from the handler
2020-04-03 01:03:11 +02:00
Viljami Kuosmanen
3742fdfb04 openapi: disable cors headers for /api/** paths
Still enabled for /rest/** and **/openapi.json
2020-04-03 01:03:11 +02:00
Viljami Kuosmanen
6aa30f213a openapi: add jsonp support + test 2020-04-03 01:03:11 +02:00
Viljami Kuosmanen
e821bbcad8 openapi: add documentation, small optimisation 2020-04-03 01:03:11 +02:00
Viljami Kuosmanen
c2cca39c7d openapi: minor improvements 2020-04-03 01:03:11 +02:00
Viljami Kuosmanen
172da1483e openapi: add commented out version of response processors
For some weird reason, these seem to be part of the original swagger
implementation but tests assume they're turned off.

Perhaps a difference between /rest and /api?
2020-04-03 01:03:11 +02:00
Viljami Kuosmanen
5792f7224a openapi: add response objects 2020-04-03 01:03:11 +02:00
Viljami Kuosmanen
03d8964a7a openapi: upgrade to openapi-backend 2.4.0 2020-04-03 01:03:11 +02:00
Viljami Kuosmanen
f9d24f5a39 openapi: remove swagger-node-express 2020-04-03 01:03:11 +02:00
Viljami Kuosmanen
25e081ac89 openapi: implement API handler with openapi-backend
- Tests pass 
- Added openapi-backend hook
- Generating OpenAPI v3 definitions for each API version
- Definitions served /api/openapi.json /api/{version}/openapi.json
2020-04-03 01:03:11 +02:00
muxator
b7ac0f7991 Settings: allow the existence of "password" and "hash" attributes for users.
This brings back compatibility with ep_hash_auth, which was inadvertently broken
with 28a3bba4c1.
Based on work by John McLear.

Fixes #3681.
2020-04-01 04:13:21 +02:00
John McLear
53f126082a SessionStore: replace password with PASSWORD_HIDDEN when storing in db
Fixes #3421.
2020-04-01 03:39:05 +02:00
muxator
4ee5ddb496 caching_midleware: also run when nodejs does not have crypto module
According to the nodejs docs [0] the `crypto` module might be unavailable on
some platforms:

> It is possible for Node.js to be built without including support for the
> crypto module. In such cases, calling require('crypto') will result in an
> error being thrown

A description of such scenarios can be found here [1].

> * running non-standard node in a resource- or security-constrained
>   environment
> * running in emulated environment (browserify, webpack etc.)
> * building node from source and omitting openssl/crypto for random reason

TypeScript guys dealt with this same issue and they resolved it in an elegant
way in [2].

We copy that approach here: if importing crypto fails at runtime, we replace
sha256 with djb2 [3], which is weaker, but works for our case.

The djb2 story is fun: see this Stack Overflow post [4], and the original
mailing list post from 1991 [5] by Daniel J. Bernstein [6].

He was 20 at the time!

[0] https://nodejs.org/docs/latest-v10.x/api/crypto.html#crypto_determining_if_crypto_support_is_unavailable
[1] https://github.com/microsoft/TypeScript/issues/19100#issuecomment-335871998
[2] 9677b0641c
[3] http://www.cse.yorku.ca/~oz/hash.html#djb2
[4] https://stackoverflow.com/questions/1579721/why-are-5381-and-33-so-important-in-the-djb2-algorithm
[5] https://groups.google.com/forum/#!msg/comp.lang.c/lSKWXiuNOAk/zstZ3SRhCjgJ
[6] https://en.wikipedia.org/wiki/Daniel_J._Bernstein
2020-03-31 04:09:49 +02:00
Tom Hudson
fc754c9a1d Switches cacheKey from base64 of path to sha256 of path; fixes #3502 2020-03-31 02:42:58 +02:00