The case where a consumer stops downloading media that is currently
being streamed is now able to be handled explicitly.
That scenario isn't really an error, it is expected behaviour.
This PR adds a custom exception which allows us to drop the log level
for this specific case from `WARNING` to `INFO`.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Eric Eastwood <erice@element.io>
This introduces a dedicated API for MAS to consume. Companion PR on the
MAS side: element-hq/matrix-authentication-service#4801
This has a few advantages over the previous admin API:
- it works on workers (this will be documented once we stabilise MSC3861
as a whole)
- it is more efficient because more focused
- it propagates trace contexts from MAS
- it is only accessible to MAS (through the shared secret) and will let
us remove the weird hack that made this token 'admin' with a ghost
'@__oidc_admin:' user
The next MAS version should support it, but will be opt-in. The version
after that should use this new API by default
---------
Co-authored-by: Eric Eastwood <erice@element.io>
Spawning from https://github.com/element-hq/synapse/pull/18689
Example CI failure that will stop people from leaving stray `Cargo.lock`
changes behind,
```
Error: Cargo.lock has uncommitted changes after install. Please run 'poetry install --extras all' and commit the Cargo.lock changes.
```
The main goal of this PR is to handle device list changes onto multiple
writers, off the main process, so that we can have logins happening
whilst Synapse is rolling-restarting.
This is quite an intrusive change, so I would advise to review this
commit by commit; I tried to keep the history as clean as possible.
There are a few things to consider:
- the `device_list_key` in stream tokens becomes a
`MultiWriterStreamToken`, which has a few implications in sync and on
the storage layer
- we had a split between `DeviceHandler` and `DeviceWorkerHandler` for
master vs. worker process. I've kept this split, but making it rather
writer vs. non-writer worker, using method overrides for doing
replication calls when needed
- there are a few operations that need to happen on a single worker at a
time. Instead of using cross-worker locks, for now I made them run on
the first writer on the list
---------
Co-authored-by: Eric Eastwood <erice@element.io>
Normal install results in `Cargo.lock` changes constantly popping up for me as I navigate my
branches. This was probably caused by some Depdendabot PR updating the
`Cargo.toml` without `Cargo.lock` or something.
```
poetry install --extras all
```
In another PR, I've also added CI to ensure we don't leave `Cargo.lock`
changes behind to avoid this annoyance in the future ->
https://github.com/element-hq/synapse/pull/18693
Clean up `MetricsResource`, Prometheus hacks
(`_set_prometheus_client_use_created_metrics`), and better document why
we care about having a separate `metrics` listener type.
These clean-up changes have been split out from
https://github.com/element-hq/synapse/pull/18584 since that PR was
closed.
Bumps [lxml](https://github.com/lxml/lxml) from 5.4.0 to 6.0.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/lxml/lxml/blob/master/CHANGES.txt">lxml's
changelog</a>.</em></p>
<blockquote>
<h1>6.0.0 (2025-06-26)</h1>
<h2>Features added</h2>
<ul>
<li>
<p>GH#463: <code>lxml.html.diff</code> is faster and provides
structurally better diffs.
Original patch by Steven Fernandez.</p>
</li>
<li>
<p>GH#405: The factories <code>Element</code> and
<code>ElementTree</code> can now be used in type hints.</p>
</li>
<li>
<p>GH#448: Parsing from <code>memoryview</code> and other buffers is
supported to allow zero-copy parsing.</p>
</li>
<li>
<p>GH#437: <code>lxml.html.builder</code> was missing several HTML5 tag
names.
Patch by Nick Tarleton.</p>
</li>
<li>
<p>GH#458: <code>CDATA</code> can now be written into the incremental
<code>xmlfile()</code> writer.
Original patch by Lane Shaw.</p>
</li>
<li>
<p>A new parser option <code>decompress=False</code> was added that
controls the automatic
input decompression when using libxml2 2.15.0 or later. Disabling this
option
by default will effectively prevent decompression bombs when handling
untrusted
input. Code that depends on automatic decompression must enable this
option.
Note that libxml2 2.15.0 was not released yet, so this option currently
has no
effect but can already be used.</p>
</li>
<li>
<p>The set of compile time / runtime supported libxml2 feature names is
available as
<code>etree.LIBXML_COMPILED_FEATURES</code> and
<code>etree.LIBXML_FEATURES</code>.
This currently includes
<code>catalog</code>, <code>ftp</code>, <code>html</code>,
<code>http</code>, <code>iconv</code>, <code>icu</code>,
<code>lzma</code>, <code>regexp</code>, <code>schematron</code>,
<code>xmlschema</code>, <code>xpath</code>, <code>zlib</code>.</p>
</li>
</ul>
<h2>Bugs fixed</h2>
<ul>
<li>
<p>GH#353: Predicates in <code>.find*()</code> could mishandle tag
indices if a default namespace is provided.
Original patch by Luise K.</p>
</li>
<li>
<p>GH#272: The <code>head</code> and <code>body</code> properties of
<code>lxml.html</code> elements failed if no such element
was found. They now return <code>None</code> instead.
Original patch by FVolral.</p>
</li>
<li>
<p>Tag names provided by code (API, not data) that are longer than
<code>INT_MAX</code>
could be truncated or mishandled in other ways.</p>
</li>
<li>
<p><code>.text_content()</code> on <code>lxml.html</code> elements
accidentally returned a "smart string"
without additional information. It now returns a plain string.</p>
</li>
<li>
<p>LP#2109931: When building lxml with coverage reporting, it now
disables the <code>sys.monitoring</code>
support due to the lack of support in <a
href="https://redirect.github.com/nedbat/coveragepy/issues/1790">nedbat/coveragepy#1790</a></p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="2a67034bc2"><code>2a67034</code></a>
Prepare release of 6.0.0.</li>
<li><a
href="e0b4e02182"><code>e0b4e02</code></a>
Update changelog.</li>
<li><a
href="d3f4dcf689"><code>d3f4dcf</code></a>
Build: Upgrade libxml2 to latest 2.14.4.</li>
<li><a
href="014e51cce3"><code>014e51c</code></a>
Build: Add Windows arm64 wheel builds (<a
href="https://redirect.github.com/lxml/lxml/issues/465">GH-465</a>)</li>
<li><a
href="d3914dcb6a"><code>d3914dc</code></a>
Only use "xmlCtxtIsStopped()" from libxml2 2.15.0 on since it
fails to cover ...</li>
<li><a
href="6e41390275"><code>6e41390</code></a>
Avoid reading the deprecated "disableSAX" attribute of
"xmlParserCtxt".</li>
<li><a
href="f85da81b1d"><code>f85da81</code></a>
Use newer "language_level=3" in ElementPath module.</li>
<li><a
href="787315eb54"><code>787315e</code></a>
Build: bump pypa/cibuildwheel in the github-actions group (<a
href="https://redirect.github.com/lxml/lxml/issues/464">#464</a>)</li>
<li><a
href="fb3adb1dce"><code>fb3adb1</code></a>
Readme: Add project income report for 2024.</li>
<li><a
href="8e61a757c8"><code>8e61a75</code></a>
Fit cached tuple more nicely into cachelines.</li>
<li>Additional commits viewable in <a
href="https://github.com/lxml/lxml/compare/lxml-5.4.0...lxml-6.0.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Fixes https://github.com/element-hq/synapse/issues/18659
This changes the Tokio runtime to be attached to the Twisted reactor.
This way, the Tokio runtime starts when the Twisted reactor starts, and
*not* when the module gets loaded.
This is important as starting the runtime on module load meant that it
broke when Synapse was started with `daemonize`/`synctl`, as forks only
retain the calling threads, breaking the Tokio runtime.
This also changes so that the HttpClient gets the Twisted reactor
explicitly as parameter instead of loading it from
`twisted.internet.reactor`
Refactor `Measure` block metrics to be homeserver-scoped (add
`server_name` label to block metrics).
Part of https://github.com/element-hq/synapse/issues/18592
### Testing strategy
#### See behavior of previous `metrics` listener
1. Add the `metrics` listener in your `homeserver.yaml`
```yaml
listeners:
- port: 9323
type: metrics
bind_addresses: ['127.0.0.1']
```
1. Start the homeserver: `poetry run synapse_homeserver --config-path
homeserver.yaml`
1. Fetch `http://localhost:9323/metrics`
1. Observe response includes the block metrics
(`synapse_util_metrics_block_count`,
`synapse_util_metrics_block_in_flight`, etc)
#### See behavior of the `http` `metrics` resource
1. Add the `metrics` resource to a new or existing `http` listeners in
your `homeserver.yaml`
```yaml
listeners:
- port: 9322
type: http
bind_addresses: ['127.0.0.1']
resources:
- names: [metrics]
compress: false
```
1. Start the homeserver: `poetry run synapse_homeserver --config-path
homeserver.yaml`
1. Fetch `http://localhost:9322/_synapse/metrics` (it's just a `GET`
request so you can even do in the browser)
1. Observe response includes the block metrics
(`synapse_util_metrics_block_count`,
`synapse_util_metrics_block_in_flight`, etc)
Fixes: #18491
Fix hotlooping due to skipped PDUs if there is still no progress to be
made.
This could bite if the event was purged since being skipped during
catch-up.
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Another config option on my quest to a `*_path` variant for every
secret. Adds the config options `recaptcha_private_key_path` and
`recaptcha_public_key_path`. Tests and docs are included.
A public key is of course no secret, but it is closely related to the
private key, so it’s still useful to have a `*_path` variant for it.
Bumps [base64](https://github.com/marshallpierce/rust-base64) from
0.21.7 to 0.22.1.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md">base64's
changelog</a>.</em></p>
<blockquote>
<h1>0.22.1</h1>
<ul>
<li>Correct the symbols used for the predefined
<code>alphabet::BIN_HEX</code>.</li>
</ul>
<h1>0.22.0</h1>
<ul>
<li><code>DecodeSliceError::OutputSliceTooSmall</code> is now
conservative rather than precise. That is, the error will only occur if
the decoded output <em>cannot</em> fit, meaning that
<code>Engine::decode_slice</code> can now be used with exactly-sized
output slices. As part of this, <code>Engine::internal_decode</code> now
returns <code>DecodeSliceError</code> instead of
<code>DecodeError</code>, but that is not expected to affect any
external callers.</li>
<li><code>DecodeError::InvalidLength</code> now refers specifically to
the <em>number of valid symbols</em> being invalid (i.e. <code>len % 4
== 1</code>), rather than just the number of input bytes. This avoids
confusing scenarios when based on interpretation you could make a case
for either <code>InvalidLength</code> or <code>InvalidByte</code> being
appropriate.</li>
<li>Decoding is somewhat faster (5-10%)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="e144006974"><code>e144006</code></a>
v0.22.1</li>
<li><a
href="64cca59ddb"><code>64cca59</code></a>
Merge pull request <a
href="https://redirect.github.com/marshallpierce/rust-base64/issues/271">#271</a>
from JobanSD/patch-1</li>
<li><a
href="838355e0ac"><code>838355e</code></a>
Correct BinHex 4.0 alphabet according to specifications</li>
<li><a
href="bf15ccf30a"><code>bf15ccf</code></a>
Merge pull request <a
href="https://redirect.github.com/marshallpierce/rust-base64/issues/270">#270</a>
from marshallpierce/mp/clippy</li>
<li><a
href="fc6aabee8a"><code>fc6aabe</code></a>
Appease clippy</li>
<li><a
href="9a518a2d5d"><code>9a518a2</code></a>
Merge pull request <a
href="https://redirect.github.com/marshallpierce/rust-base64/issues/267">#267</a>
from bdura/patch-1</li>
<li><a
href="d96c80f242"><code>d96c80f</code></a>
Merge branch 'marshallpierce:master' into patch-1</li>
<li><a
href="5d70ba7576"><code>5d70ba7</code></a>
Merge pull request <a
href="https://redirect.github.com/marshallpierce/rust-base64/issues/269">#269</a>
from marshallpierce/mp/decode-precisely</li>
<li><a
href="efb6c006c7"><code>efb6c00</code></a>
Release notes</li>
<li><a
href="2b91084a31"><code>2b91084</code></a>
Add some tests to boost coverage</li>
<li>Additional commits viewable in <a
href="https://github.com/marshallpierce/rust-base64/compare/v0.21.7...v0.22.1">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
`commonmark` has been deprecated in favor of `markdown-it-py`, and its
type hints have been [removed from
typeshed](https://github.com/python/typeshed/issues/13725).
This switches the release script to use `markdown-it-py` instead of
`commonmark` for parsing the `CHANGES.md`
