Modify SSL backlog buffer from LIFO to queue/FIFO
If data ends up in the ssl_backlog_buffer because we are waiting for a
handshake to finish, then this change ensures that the data is sent out
in the proper order once the handshake completes. Previous code was
sending in LIFO order.
Fix the issue https://github.com/coturn/coturn/issues/118 that prevent
the process to start as coturn never send the notification to systemd
thus preventing the startup.
Cherry-picked from #990 thanks to @rapsys
Rewriting openssl initialization code (threading support to make it
cleaner
- Regroup functions so that there is one ifdef (for old code and new
code)
- Modern openssl (>1.0.2) does not need any synchornization routines so
they are empty
- Old openssl (<=1.0.2) now require `OPENSSL_THREADS` which allows
running multiple threads in turnserver. Not having turnserver
multi-threaded is a huge waste. `OPENSSL_THREADS` is now a requirement.
Test Plan:
- CI builds pass for openssl versions 1.0.2, 1.1.1, 3.0, including tests
Adding fuzzing to finding memory-corruption-related bugs.
Hello coturn team,
Can you check this pr harness suite for creating harnesses and compiling
harnesses?
Any other thoughts on adding a new interface for fuzzing support ?
Signed-off-by: 0x34d <ajsinghyadav00@gmail.com>
Signed-off-by: 0x34d <ajsinghyadav00@gmail.com>
Essentially, for a DTLS client (that we haven't heard from before), the code in handle_udp_packet will have created the chs/ioa_socket in the block just above my change (see dtls_server_input_handler's call to dtls_accept_client_connection that calls create_ioa_socket_from_ssl). This only happens if the first message received from a client is a DTLS handshake. Otherwise, we have received UDP data from a new endpoint that is not a DTLS handshake, so it is raw UDP and the code just below my if statement will have created a UDP_SOCKET in the create_ioa_socket_from_fd call, allowing further processing of the RAW UDP.
This was tested by trying to perform a TURN allocation via UDP (not DTLS) when no-udp setting was enabled.
Store sanitized version of DB connection string with password masked
(replace all chars with * which exposes its length)
Use sanitized version when logging connection string
Fixes#1017 and #272
turnserver includes support for SCTP and tries to initialize listener
sockets with SCTP protocol. On machines where SCTP definitions exist but
the protocol is not provided - socket() returns error which shows up as
`socket: protocol not supported`
This change improves a few related pieces of code:
- Log error instead of perror
- config script detect sctp.h and if not present - defines TURN_NO_SCTP
- CMake fully disables SCTP (for now - requires custom module to detect
SCTP presence)
Fixes#492
Hello,
while using the `redis-statsdb` option, I found that coturn is leaking
sockets (and memory) in case of redis reconnection.
This occurs a lot to me, because in my setup I have a `coturn -> haproxy
-> redis` and if all my redis servers are down, HaProxy abruptly close
the connection to coturn and coturn reconnects periodically. After some
time I can see thousands of pending sockets (`CLOSE_WAIT`) :
```
user@server[11:32:48 UTC]:~$ sudo lsof -i | grep turn
turnserve 461797 root 15u IPv4 12856075 0t0 TCP server:3478 (LISTEN)
turnserve 461797 root 22u IPv4 12856081 0t0 TCP server:3478 (LISTEN)
turnserve 461797 root 23u IPv4 12857384 0t0 UDP server:3478
turnserve 461797 root 24u IPv4 12857385 0t0 UDP server:3478
turnserve 461797 root 36u IPv4 12857390 0t0 TCP server:5766 (LISTEN)
turnserve 461797 root 43u IPv4 12856096 0t0 TCP server:10059->haproxy-server:redis (CLOSE_WAIT)
turnserve 461797 root 46u IPv4 12857403 0t0 TCP server:10087->haproxy-server:redis (CLOSE_WAIT)
turnserve 461797 root 48u IPv4 12856124 0t0 TCP server:53867->haproxy-server:redis (CLOSE_WAIT)
turnserve 461797 root 50u IPv4 12857633 0t0 TCP server:53875->haproxy-server:redis (CLOSE_WAIT)
turnserve 461797 root 51u IPv4 12856138 0t0 TCP server:53877->haproxy-server:redis (CLOSE_WAIT)
turnserve 461797 root 54u IPv4 12857738 0t0 TCP server:10001->haproxy-server:redis (CLOSE_WAIT)
turnserve 461797 root 55u IPv4 12856152 0t0 TCP server:10003->haproxy-server:redis (CLOSE_WAIT)
.... (many many lines)
```
After searching and using valgrind I found 2 interesting leaks:
```
...
==460721== 32 bytes in 1 blocks are definitely lost in loss record 586 of 1,053
==460721== at 0x483877F: malloc (vg_replace_malloc.c:307)
==460721== by 0x1414FF: RyconninfoParse (dbd_redis.c:69)
==460721== by 0x141B04: get_redis_async_connection (dbd_redis.c:169)
==460721== by 0x110D7B: create_ioa_engine (ns_ioalib_engine_impl.c:407)
==460721== by 0x12ECB0: setup_admin_thread (turn_admin_server.c:1309)
==460721== by 0x127724: run_admin_server_thread (netengine.c:1815)
==460721== by 0x4DA9EA6: start_thread (pthread_create.c:477)
==460721== by 0x4EC0AEE: clone (clone.S:95)
...
==460979== 2,170 (688 direct, 1,482 indirect) bytes in 2 blocks are definitely lost in loss record 1,029 of 1,049
==460979== at 0x483AD7B: realloc (vg_replace_malloc.c:834)
==460979== by 0x49A1BD0: ??? (in /usr/lib/x86_64-linux-gnu/libhiredis.so.0.14)
==460979== by 0x49A2829: redisAsyncConnect (in /usr/lib/x86_64-linux-gnu/libhiredis.so.0.14)
==460979== by 0x13DB37: redis_reconnect (hiredis_libevent2.c:331)
==460979== by 0x13D1A7: redisLibeventReadEvent (hiredis_libevent2.c:101)
==460979== by 0x4D5135E: ??? (in /usr/lib/x86_64-linux-gnu/libevent_core-2.1.so.7.0.1)
==460979== by 0x4D51A9E: event_base_loop (in /usr/lib/x86_64-linux-gnu/libevent_core-2.1.so.7.0.1)
==460979== by 0x126D5A: run_events (netengine.c:1579)
==460979== by 0x127272: run_general_relay_thread (netengine.c:1707)
==460979== by 0x4DA9EA6: start_thread (pthread_create.c:477)
==460979== by 0x4EC0AEE: clone (clone.S:95)
==460979==
...
```
I made 1 commit for each fix.
Obviously with these fixes, I don't have anymore the leaks of thousands
of sockets (even after some time)
Thanks & hope it helps.
Thibaut
```
==6418==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x4e7530 in bcmp /src/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc:906:10
#1 0x55463d in stun_check_message_integrity_by_key_str coturn/src/client/ns_turn_msg.c:1989:5
#2 0x554acc in stun_check_message_integrity_str coturn/src/client/ns_turn_msg.c:2008:9
#3 0x5358c0 in LLVMFuzzerTestOneInput coturn/fuzz/FuzzStun.c:37:5
#4 0x43ede3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15
#5 0x42a542 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
#6 0x42fdec in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:860:9
#7 0x459322 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
#8 0x7f4cb21790b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/libc-start.c:308:16
#9 0x42070d in _start
Uninitialized value was created by an allocation of 'new_hmac' in the stack frame of function 'stun_check_message_integrity_by_key_str'
#0 0x5538c0 in stun_check_message_integrity_by_key_str coturn/src/client/ns_turn_msg.c:1927
```
Split out work on libtelnet from #855
Required to update libtelnet to introduce compatibility with Windows
This change contains vanilla code of
[libtelnet](https://github.com/seanmiddleditch/libtelnet) v0.23 (latest
tag)
Test Plan:
- run turnserver locally, set cli password, connect to the turnserver
cli interface
- run a few commands - get output
This code is in as back as git can see. Removed for now as it has no use
at all.
Also reduces traffic to redis (though will not reduce any load on redis)
Refs #150
Using clang-tidy to detect unused header files
Inspired by #855
Test Plan:
- Rebuild all on mac, review no warnings/errors
- Pass builds/docker build - review for no issues
Similar to #989, use a single SSL context for all versions of DTLS
protocol
- Add support for modern API (protocol version independent APIs)
- Add DTLS test to the CI test
- Removing calls to `SSL_CTX_set_read_ahead` in DTLS context (does
nothing as DTLS is datagram protocol - we always get the whole datagram
so this call has no impact)
Fixes#924
Hello, and thank you for this great project!
When sending a mobility refresh request, the server is responding with
an invalid stun packet. Specifically, only the last 24 bytes generated
for message integrity are sent.
Request:
Response:
This appears to be caused by the buffer length not being updated when
`STUN_ATTRIBUTE_SOFTWARE` is added to the response, as it's being
assigned to a `len` variable scoped to that conditional.
As a result, the next call to `stun_get_command_message_len_str` when
handling message integrity returns `-1` and resizes the buffer, etc.
Passing in the original `len` to `stun_attr_add_str` for attribute
software fixes the issue, but apologies if this is not the right way to
fix this!
STRCPY macro makes pointer comparison which creates a warning
In those places, replace the macro with `strncpy` with careful review of
destination buffer size
With this change I do not get compiler warnings at all
Install dependencies and build with prometheus support during CI builds
Ubuntu 16.04 has issues supporting `MHD_USE_EPOLL_INTERNAL_THREAD` so
disabling it (using SELECT)
Should not impact anything as the only reason we use Ubuntu 16.04 is to
validate build against openssl-1.0.2
Fixes #998
openssl allows multiple TLS version support through a single SSL_CTX
object.
This PR replaces 4 per-version SSL_CTX objects with a single object
(DTLS is not yet changed).
SSL context initialization code for openssl with modern API (>=1.1.0)
uses `TLS_server_method` and `SSL_CTX_set_min_proto_version` instead of
enabling specific TLS version. Byproduct of this is TLSv1_3 support when
used with openssl-1.1.1 and above
TLS 1.2 and TLS 1.3 cannot be disabled (as before)
Test plan:
- run_tests.sh script now runs turnserver with SSL certificate (which
enables TLS support)
- run_tests.sh now has one more basic test that uses TLS protocol
Co-authored-by: Pavel Punsky <pavel.punsky@epicgames.com>
Before:
```
./bin/turnserver --prometheus -m 32
...
Socket descriptor larger than FD_SETSIZE: 2660 > 1024
0: : ERROR: Could not start Prometheus collector!
```
After:
```
./bin/turnserver --prometheus -m 32
...
0: : Prometheus collector started successfully.
```
Also added `MHD_USE_ERROR_LOG` so the actual problems during startup of
promhttp are printed. Without it the `Socket descriptor larger than
FD_SETSIZE: 2660 > 1024` error was not visible and finding the cause of
the startup problems of promhttp was not obvious at first.
Replace all instances of `bzero` with memset by find-replace-edit.
This is straightforward replacement which is suboptimal in a few cases
(for example we could use calloc instead of malloc+memset(0))
Inspired by #855
There are too many defines that are, eventually, used in one place so
just inlining.
Current code generates following warning:
```
warning: macro expansion producing 'defined' has undefined behavior [-Wexpansion-to-defined]
```
With the fix there is no warning
Co-authored-by: Pavel Punsky <pavel.punsky@epicgames.com>
openssl-3.0 deprecated some APIs and introduced new APIs instead:
`SSL_get_peer_certificate ` -> `SSL_get1_peer_certificate `
`FIPS_mode()`->`EVP_default_properties_is_fips_enabled()`
`EVP_MD_CTX_set_flags()`->`EVP_default_properties_enable_fips()`
specifically for enabling FIPS mode
This change should workaround that by ifdef-ing old/new versions of
openssl and APIs - so pre-3.0 use existing APIs (so not change there)
and >=3.0 will use new APIs (whether it actually works or not is still
TBD as this is just a first step in openssl-3.0 support)
Should fix#886
Test Plan:
Run CI build that supports ubuntu-20.04 (openssl-1.1.1) and ubuntu-22.04
(openssl-3.0.2)
Both builds pass
None of them have FIPS support (which for 1.1.x stays the same as
before)
Co-authored-by: Pavel Punsky <pavel.punsky@epicgames.com>
- Preserve file timestamps when using `install`
- Use permissions `0644` rather than default `0755` for installing man
pages
The alternative calls of `cp` are using `cp -p` as well (if `install` is
unavailable).
`SSL_OP_NO_RENEGOTIATION` is only supported in openssl-1.1.0 and above
Older versions have `SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS `
Fixes#977 and #952
Test:
Build in a docker container running running openssl-1.0.2g (ubuntu
16.04) successfully (without the fix getting the same errors)
- omit publishing and releasing Docker images in forks
- verify Git tag to match Makefile before releasing
- auto-cancel concurrent CI pipelines for the same Git ref
`vsnprintf` will stop at the max buffer size as provided in its 2nd
argument
But the return value is `The number of characters that would have been
written if n had been sufficiently large` meaning it can be larger than
actual buffer size
`fwrite` will actually use the larger, incorrect number and dump
unrelated memory to log (and crash with high confidence)
Test:
- Query admin interface with super long path (>16KB) - crash
- With the fix - no crash with the same input, log line cut off
Co-authored-by: Pavel Punsky <pavel.punsky@epicgames.com>
Writing outside of a buffer can only happen if incoming HTTP request is longer than UDP_STUN_BUFFER_SIZE (16KB).
This change validates that the request is no longer than the buffer size and drops it if it is the case
Fixes#342
Test plan:
- Run in debugger and send a 16KB request using curl - response returns, logs correct
- Send 16KB + 1b request - warning logged and request dropped
Co-authored-by: Pavel Punsky <pavel.punsky@epicgames.com>
When using `turnutils_uclient` with `-S` flag (TLS or DTLS) it is not required to load certificates. Only load certificates when corresponding flags are provided
Fixes#376 which prevented using `turnutils_uclient` for testing TLS/DTLS connections
Test plan:
- Run local turnserver with certificates `./bin/turnserver --cert ./bin/public.pem --pkey ./bin/private.key --use-auth-secret --static-auth-secret=secret --realm=north.gov --allow-loopback-peers --no-cli --verbose`
- Run fixed uclient without TLS/DTLS`./bin/turnutils_uclient -e 127.0.0.1 -X -g -u user -W secret 127.0.0.1` and get success result (just to make sure non-secure still works)
- Run fixed uclient with TLS `./bin/turnutils_uclient -e 127.0.0.1 -X -g -u user -W secret -t -S 127.0.0.1` and get success result
- Run fixed uclient with DTLS `./bin/turnutils_uclient -e 127.0.0.1 -X -g -u user -W secret -S 127.0.0.1` and get success result
- Run unpatched uclient with TLS `./bin/turnutils_uclient -e 127.0.0.1 -X -g -u user -W secret -t -S 127.0.0.1` - error about missing certificate files
Co-authored-by: Pavel Punsky <pavel.punsky@epicgames.com>
Actually, as below, before fixing, the sessions_map may be larger and
larger when register_callback_on_ioa_socket inside
open_client_connection_session retrun -1. The reason is that, if
register_callback_on_ioa_socket return -1, no read/write/ event will be
registered and no timeout event will be registered, in this case, the
session will never be deleted from sessions_map.
After fixing, when register_callback_on_ioa_socket fail，it doesn't
return immediately, instead, we still register
client_to_be_allocated_timeout_handler in the next step of
open_client_connection_session. In this case, the unuseful session will
be deleted and freed after timeout.
@eakraly @ggarber
