diff --git a/.gitignore b/.gitignore index 244b73a9384..546eab8f656 100644 --- a/.gitignore +++ b/.gitignore @@ -71,6 +71,8 @@ source/developer/localization.md *.crt .aider* +**/.claude/settings.local.json + # Sphinx build artifacts *.doctree _build/ @@ -78,4 +80,4 @@ build/ # Cursor instructions *.cursor -eng.md \ No newline at end of file +eng.md diff --git a/source/_generated/agents/docs/admin_guide.md b/source/_generated/agents/docs/admin_guide.md index b6706abc4e5..b86fc4d5a67 100644 --- a/source/_generated/agents/docs/admin_guide.md +++ b/source/_generated/agents/docs/admin_guide.md @@ -471,7 +471,7 @@ The MCP server provides the following tools to AI agents and external clients: ### Deployment -![MCP Server Configuration](img/system-console-mcp.png) +![MCP Server Configuration](/agents/docs/img/system-console-mcp.png) #### For AI Agents diff --git a/source/administration-guide/configure/calls-deployment.md b/source/administration-guide/configure/calls-deployment.md index 8d0131a58da..154e6afa6f3 100644 --- a/source/administration-guide/configure/calls-deployment.md +++ b/source/administration-guide/configure/calls-deployment.md @@ -1,30 +1,45 @@ -# Calls self-hosted deployment +# Deploy Mattermost Calls ```{include} ../../_static/badges/all-commercial.md ``` -Mattermost Calls is an excellent option for organizations demanding enhanced security and control over their communication infrastructure. Calls is designed to operate securely in self-hosted deployments, including [air-gapped environments](https://docs.mattermost.com/configure/calls-deployment.html#air-gapped-deployments), ensuring private communication without reliance on public internet connectivity with flexible configuration options for complex network requirements. +This document provides an overview of Mattermost Calls deployment options for self-hosted environments, including [air-gapped environments](https://docs.mattermost.com/configure/calls-deployment.html#air-gapped-deployments), ensuring private communication without reliance on public internet connectivity with flexible configuration options for complex network requirements. -This document provides information on how to successfully make the Calls plugin work on self-hosted deployments. It also outlines some of the most common deployment strategies with example diagrams, and provides the deployment guidelines for the recording, transcription, and live captions service. -## Terminology +## Quick Links + +For detailed information on specific topics, please refer to these specialized guides: -- [WebRTC](https://bloggeek.me/webrtcglossary/webrtc-2/): The set of underlying protocols/specifications on top of which calls are implemented. -- **RTC (Real Time Connection)**: The real-time connection. This is the channel used to send media tracks (audio/video/screen). -- **WS (WebSocket)**: The WebSocket connection. This is the channel used to set up a connection (signaling process). -- [NAT (Network Address Translation)](https://bloggeek.me/webrtcglossary/nat/): A networking technique to map IP addresses. -- [STUN (Session Traversal Utilities for NAT)](https://bloggeek.me/webrtcglossary/stun/): A protocol/service used by WebRTC clients to help traversing NATs. On the server side it's mainly used to figure out the public IP of the instance. -- [TURN (Traversal Using Relays around NAT)](https://bloggeek.me/webrtcglossary/turn/): A protocol/service used to help WebRTC clients behind strict firewalls connect to a call through media relay. +- {doc}`RTCD Setup and Configuration `: Comprehensive guide for setting up the dedicated RTCD service +- {doc}`Calls Offloader Setup and Configuration `: Comprehensive guide for setting up the calls-offloader service for recording and transcription +- {doc}`Calls Troubleshooting `: Detailed troubleshooting steps and debugging techniques +- {doc}`Calls Metrics and Monitoring `: Guide to monitoring Calls performance using metrics and observability +- {doc}`Calls Deployment on Kubernetes `: Detailed guide for deploying Calls in Kubernetes environments -## Plugin components +## About Mattermost Calls -- **Calls plugin**: This is the main entry point and a requirement to enable channel calls. +Mattermost Calls provides integrated audio calling and screen sharing capabilities within Mattermost channels. It's built on WebRTC technology and can be deployed either: -- **rtcd**: This is an optional service that can be deployed to offload all the functionality and data processing involved with the WebRTC connections. Read more about when and why to use [rtcd](#the-rtcd-service) below. +1. **Integrated mode**: Built into the Calls plugin (simpler, suitable for smaller deployments) +2. **RTCD mode**: Using a dedicated service for improved performance and scalability (recommended for production environments) + +## Terminology -## Requirements +- [WebRTC](https://bloggeek.me/webrtcglossary/webrtc-2/): The set of protocols on which calls are built +- **RTC**: Real-Time Connection channel used for media (audio/video/screen) +- **WS**: WebSocket connection used for signaling and connection setup +- **SFU**: Selective Forwarding Unit, routes media between participants +- [NAT](https://bloggeek.me/webrtcglossary/nat/): Network Address Translation for mapping IP addresses +- [STUN](https://bloggeek.me/webrtcglossary/stun/): Protocol used by WebRTC clients to help traverse NATs +- [TURN](https://bloggeek.me/webrtcglossary/turn/): Protocol to relay media for clients behind strict firewalls -### Server +## Key Components + +- **Calls plugin**: The main plugin that enables calls functionality. Installed by default in Mattermost self-hosted deployments. +- **RTCD service**: Optional dedicated service for offloading media processing (Enterprise feature). Typically deployed to dedicated servers or containers. See [RTCD Setup and Configuration](calls-rtcd-setup.md) for details. +- **calls-offloader**: Service for call recording and transcription (if enabled). Typically deployed to dedicated servers. See [Calls Offloader Setup and Configuration](calls-offloader-setup.md) for setup and troubleshooting details. + +## Network Requirements - Run Mattermost server on a secure (HTTPs) connection. This is a necessary requirement on the client to allow capturing devices (e.g., microphone, screen). See the [config TLS](https://docs.mattermost.com/deploy/server/setup-tls.html) section for more info. - See [network requirements](#network) below. @@ -88,7 +103,7 @@ This document provides information on how to successfully make the Calls plugin RTC (Calls plugin or rtcd) 8443 UDP (incoming) -Mattermost clients (Web/Desktop/Mobile) +Mattermost clients (Web/Desktop/Mobile) and jobs spawned by Calls Offloader (Recorder, Transcriber) Mattermost instance or rtcd service To allow clients to establish connections that transport calls related media (e.g. audio, video). This should be open on any network component (e.g. NAT, firewalls) in between the instance running the plugin (or rtcd) and the clients joining calls so that UDP traffic is correctly routed both ways (from/to clients). @@ -96,7 +111,7 @@ This document provides information on how to successfully make the Calls plugin RTC (Calls plugin or rtcd) 8443 TCP (incoming) -Mattermost clients (Web/Desktop/Mobile) +Mattermost clients (Web/Desktop/Mobile) and jobs spawned by Calls Offloader (Recorder, Transcriber) Mattermost instance or rtcd service To allow clients to establish connections that transport calls related media (e.g. audio, video). This should be open on any network component (e.g. NAT, firewalls) in between the instance running the plugin (or rtcd) and the clients joining calls so that TCP traffic is correctly routed both ways (from/to clients). This can be used as a backup channel in case clients are unable to connect using UDP. It requires rtcd version >= v0.11 and Calls version >= v0.17. @@ -119,13 +134,7 @@ This document provides information on how to successfully make the Calls plugin -#### Air-gapped deployments - -Mattermost Calls can function in air-gapped environments. Exposing Calls to the public internet is only necessary when users need to connect from outside the local network, and no existing method supports that connection. In such setups: - -- Users should connect from within the private/local network. This can be done on-premises, through a VPN, or via virtual machines. -- Configuring a STUN server is unnecessary, as all connections occur within the local network. -- The [ICE Host Override](https://docs.mattermost.com/configure/plugins-configuration-settings.html#ice-host-override) configuration setting can be optionally set with a local IP address (e.g., 192.168.1.45), depending on the specific network configuration and topology. +For complete network requirements, see the [RTCD Setup and Configuration](calls-rtcd-setup.md) guide. ### RHEL deployments with firewalld and fapolicyd @@ -151,16 +160,16 @@ The Calls plugin runs as part of Mattermost and is covered by the standard Matte - All Mattermost customers can start, join, and participate in 1:1 audio calls with optional screen sharing. - For group calls up to 50 concurrent users, Mattermost Enterprise, Professional, or Mattermost Cloud is required. -- Enterprise customers can also [record calls](https://docs.mattermost.com/collaborate/make-calls.html#record-a-call), enable [live text captions](https://docs.mattermost.com/collaborate/make-calls.html#live-captions-during-calls) during calls, and [transcribe recorded calls](https://docs.mattermost.com/collaborate/make-calls.html#transcribe-recorded-calls). We recommend that Enterprise self-hosted customers looking for group calls beyond 50 concurrent users consider using the [dedicated rtcd service](#the-rtcd-service). -- For Mattermost self-hosted deployments, System admins need to enable and configure the plugin [using the System Console](https://docs.mattermost.com/configure/plugins-configuration-settings.html#calls). The default maximum number of participants is unlimited; however, we recommend a maximum of 50 participants per call. Maximum call participants is configurable by going to **System Console > Plugin Management > Calls > Max call participants**. Call participant limits greatly depends on instance resources. For more details, refer to the [performance section](#performance) below. +- Enterprise customers can also [record calls](https://docs.mattermost.com/collaborate/make-calls.html#record-a-call), enable [live text captions](https://docs.mattermost.com/collaborate/make-calls.html#live-captions-during-calls) during calls, and [transcribe recorded calls](https://docs.mattermost.com/collaborate/make-calls.html#transcribe-recorded-calls). We recommend that Enterprise self-hosted customers looking for group calls beyond 50 concurrent users consider using the [dedicated RTCD service](#when-to-use-rtcd). +- For Mattermost self-hosted deployments, System admins need to enable and configure the plugin [using the System Console](https://docs.mattermost.com/configure/plugins-configuration-settings.html#calls). The default maximum number of participants is unlimited; however, we recommend a maximum of 50 participants per call. Maximum call participants is configurable by going to **System Console > Plugin Management > Calls > Max call participants**. Call participant limits greatly depends on instance resources. For more details, refer to the [Performance Considerations](#performance-considerations) section below. ## Configuration -For Mattermost self-hosted customers, the calls plugin is pre-packaged, installed, and enabled. Configuration to allow end-users to use it can be found in the [System Console](https://docs.mattermost.com/configure/plugins-configuration-settings.html#calls). +For Mattermost self-hosted customers, the calls plugin is pre-packaged, installed, and enabled. Configuration to allow end-users to use it can be found in ``System Console > Plugins > Calls``. ## Modes of operation -Depending on how the Mattermost server is running, there are several modes under which the Calls plugin can operate. Please refer to the section below on [the rtcd service](#the-rtcd-service) to learn about the `rtcd` and the Selective Forwarding Unit (SFU). +Depending on how the Mattermost server is running, there are several modes under which the Calls plugin can operate. Please refer to the section below on [the RTCD service](#when-to-use-rtcd) to learn about the `rtcd` and the Selective Forwarding Unit (SFU). | Mattermost deployment | SFU | SFU deployment | |-----------------------|-----|----------------| @@ -175,7 +184,7 @@ Depending on how the Mattermost server is running, there are several modes under This is the default mode when first installing the plugin on a single Mattermost instance setup. The WebRTC service is integrated in the plugin itself and runs alongside the Mattermost server. -![A diagram of the integrated configuration model of a single instance.](/images/calls-deployment-image3.png) +![A diagram of the integrated configuration model of a single instance.](../../images/calls-deployment-image3.png) #### rtcd @@ -184,7 +193,7 @@ This is the default mode when first installing the plugin on a single Mattermost An external, dedicated and scalable WebRTC service (`rtcd`) is used to handle all calls media routing. -![A diagram of a Web RTC deployment configuration.](/images/calls-deployment-image7.png) +![A diagram of a Web RTC deployment configuration.](../../images/calls-deployment-image7.png) ### High availability cluster-based @@ -195,580 +204,117 @@ An external, dedicated and scalable WebRTC service (`rtcd`) is used to handle al This is the default mode when running the plugin in a high availability cluster-based deployment. Every Mattermost node will run an instance of the plugin that includes a WebRTC service. Calls are distributed across all available nodes through the existing load-balancer: a call is hosted on the instance where the initiating websocket connection (first client to join) is made. A single call will be hosted on a single cluster node. -![A diagram of a clustered calls deployment.](/images/calls-deployment-image5.png) +![A diagram of a clustered calls deployment.](../../images/calls-deployment-image5.png) #### rtcd (High Availability) -![A diagram of an rtcd deployment.](/images/calls-deployment-image2.png) - -## Performance - -Calls performance primarily depends on two resources: CPU and bandwidth (both network latency and overall throughput). The final consumption exhibits quadratic growth with the number of clients transmitting and receiving media. - -As an example, a single call with 10 participants of which two are unmuted (transmitting voice data) will generally consume double the resources than the same call with a single participant unmuted. What ultimately counts towards performance is the overall number of concurrent media flows (in/out) across the server. - -### Benchmarks - -Here are the results from internally conducted performance and ceiling tests on a dedicated `rtcd` instance: - -#### Deployment specifications - -- 1x r6i.large nginx proxy -- 3x c5.large MM app nodes (HA) -- 2x db.x2g.xlarge RDS Aurora MySQL v8 (one writer, one reader) -- 1x (c7i.xlarge, c7i.2xlarge, c7i.4xlarge) RTCD -- 2x c7i.2xlarge load-test agents - -#### App specifications - -- Mattermost v9.6 -- Mattermost Calls v0.28.0 -- RTCD v0.16.0 -- load-test agent v0.28.0 - -#### Media specifications - -- Speech sample bitrate: 80Kbps -- Screen sharing sample bitrate: 1.6Mbps - -#### Results - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
CallsParticipants/callUnmuted/callScreen sharingCPU (avg)Memory (avg)Bandwidth (in/out)Instance type (RTCD)
110002no47%1.46GB1Mbps / 194Mbpsc7i.xlarge
18001yes64%1.43GB2.7Mbps / 1.36Gbpsc7i.xlarge
110001yes79%1.54GB2.9Mbps / 1.68Gbpsc7i.xlarge
101001yes74%1.56GB18.2Mbps / 1.68Gbpsc7i.xlarge
100102no49%1.46GB18.7Mbps / 175Mbpsc7i.xlarge
100101yes84%1.73GB171Mbps / 1.53Gbpsc7i.xlarge
110002no20%1.44GB1.4Mbps / 194Mbpsc7i.2xlarge
110002yes49%1.53GB3.6Mbps / 1.79Gbpsc7i.2xlarge
210001yes73%2.38GB5.7Mbps / 3.06Gbpsc7i.2xlarge
100102yes60%1.74GB181Mbps / 1.62Gbpsc7i.2xlarge
150101yes72%2.26GB257Mbps / 2.30Gbpsc7i.2xlarge
150102yes79%2.34GB271Mbps / 2.41Gbpsc7i.2xlarge
250102no58%2.66GB47Mbps / 439Mbpsc7i.2xlarge
100022no78%2.31GB178Mbps / 195Mbpsc7i.2xlarge
210002yes41%2.6GB7.23Mbps / 3.60Gbpsc7i.4xlarge
310002yes63%3.53GB10.9Mbps / 5.38Gbpsc7i.4xlarge
410002yes83%4.40GB14.5Mbps / 7.17Gbpsc7i.4xlarge
250102yes79%3.49GB431Mbps / 3.73Gbpsc7i.4xlarge
50022yes71%2.54GB896Mbps / 919Mbpsc7i.4xlarge
- -```{note} -- The tests focused on a single, vertically scaled RTCD instance to understand the processing limits within a single node. Scaling the RTCD service horizontally should be sufficient to support a higher number of calls. -- RTCD processes were executed with all performance profiling enabled (including block and mutex). This resulted in some computational overhead. -- Both speech and screen samples have slightly higher bitrates than the average produced by a real client (e.g., a browser). This gives us some safety margin over real-world deployments. -``` - -### Dedicated service - -For Enterprise customers we offer a way to offload performance costs through a [dedicated service](https://github.com/mattermost/rtcd) that can be used to further scale up calls. - -### Load testing - -We provide a [load-test tool](https://github.com/mattermost/mattermost-plugin-calls/tree/main/lt) that can be used to simulate and measure the performance impact of calls. - -### Monitoring - -Both the plugin and the external `rtcd` service expose some Prometheus metrics to monitor performance. We provide an [official dashboard](https://grafana.com/grafana/dashboards/23225-mattermost-calls-performance-monitoring/) that can be imported in Grafana. You can refer to [Performance monitoring](https://docs.mattermost.com/scale/deploy-prometheus-grafana-for-performance-monitoring.html) for more information on how to set up Prometheus and visualize metrics through Grafana. - -#### Calls plugin metrics - -Metrics for the calls plugin are exposed through the `/plugins/com.mattermost.calls/metrics` subpath under the existing Mattermost server metrics endpoint. This is controlled by the [Listen address for performance](https://docs.mattermost.com/configure/environment-configuration-settings.html#listen-address-for-performance) configuration setting. It defaults to port ``8067``. - -```{note} -- The [Metrics plugin](https://docs.mattermost.com/scale/collect-performance-metrics.html) collects application-level metrics only and does not make system or OS-level calls. As a result, data typically derived from system-level metrics may be missing in the Grafana panel. -- On Mattermost versions prior to v9.5, plugin metrics were exposed through the public `/plugins/com.mattermost.calls/metrics` API endpoint controlled by the [Web server listen address](https://docs.mattermost.com/configure/environment-configuration-settings.html#web-server-listen-address) configuration setting. This defaults to port `8065`. -``` - -**Process** - -- `mattermost_plugin_calls_process_cpu_seconds_total`: Total user and system CPU time spent in seconds. -- `mattermost_plugin_calls_process_max_fds`: Maximum number of open file descriptors. -- `mattermost_plugin_calls_process_open_fds`: Number of open file descriptors. -- `mattermost_plugin_calls_process_resident_memory_bytes`: Resident memory size in bytes. -- `mattermost_plugin_calls_process_virtual_memory_bytes`: Virtual memory size in bytes. - -**WebRTC connection** - -- `mattermost_plugin_calls_rtc_conn_states_total`: Total number of RTC connection state changes. -- `mattermost_plugin_calls_rtc_errors_total`: Total number of RTC errors. -- `mattermost_plugin_calls_rtc_rtp_bytes_total`: Total number of sent/received RTP packets in bytes. - - - Note: removed as of v0.16.0 - -- `mattermost_plugin_calls_rtc_rtp_packets_total`: Total number of sent/received RTP packets. - - - Note: removed as of v0.16.0 - -- `mattermost_plugin_calls_rtc_rtp_tracks_total`: Total number of incoming/outgoing RTP tracks. - - - Note: added as of v0.16.0 - -- `mattermost_plugin_calls_rtc_sessions_total`: Total number of active RTC sessions. - -**Application** - -- `mattermost_plugin_calls_app_handlers_time_bucket`: Time taken to execute app handlers. - - - `mattermost_plugin_calls_app_handlers_time_sum` - - - `mattermost_plugin_calls_app_handlers_time_count` - -**Database** - -- `mattermost_plugin_calls_store_ops_total`: Total number of db store operations. -- `mattermost_plugin_calls_store_methods_time_bucket`: Time taken to execute store methods. +![A diagram of an rtcd deployment.](../../images/calls-deployment-image2.png) - - `mattermost_plugin_calls_store_methods_time_sum` - - - `mattermost_plugin_calls_store_methods_time_count` -- `mattermost_plugin_calls_cluster_mutex_grab_time_bucket`: Time taken to grab global mutexes. - - - `mattermost_plugin_calls_cluster_mutex_grab_time_sum` - - - `mattermost_plugin_calls_cluster_mutex_grab_time_count` -- `mattermost_plugin_calls_cluster_mutex_locked_time_bucket`: Time spent locked in global mutexes. - - - `mattermost_plugin_calls_cluster_mutex_locked_time_sum` - - - `mattermost_plugin_calls_cluster_mutex_locked_time_count` - -**WebSocket** - -- `mattermost_plugin_calls_websocket_connections_total`: Total number of active WebSocket connections. -- `mattermost_plugin_calls_websocket_events_total`: Total number of WebSocket events. - -**Jobs** - -- `mattermost_plugin_calls_jobs_live_captions_new_audio_len_ms_bucket`: Duration (in ms) of new audio transcribed for live captions. - - - `mattermost_plugin_calls_jobs_live_captions_new_audio_len_ms_sum` - - - `mattermost_plugin_calls_jobs_live_captions_new_audio_len_ms_count` -- `mattermost_plugin_calls_jobs_live_captions_pktPayloadCh_buf_full`: Total packets of audio data dropped due to full channel. -- `mattermost_plugin_calls_jobs_live_captions_window_dropped`: Total windows of audio data dropped due to pressure on the transcriber. - -#### WebRTC service metrics - -Metrics for the `rtcd` service are exposed through the `/metrics` API endpoint under the `rtcd` API listener controlled by the `api.http.listen_address` configuration setting. It defaults to port `8045`. - -**Process** - -- `rtcd_process_cpu_seconds_total`: Total user and system CPU time spent in seconds. -- `rtcd_plugin_calls_process_max_fds`: Maximum number of open file descriptors. -- `rtcd_plugin_calls_process_open_fds`: Number of open file descriptors. -- `rtcd_plugin_calls_process_resident_memory_bytes`: Resident memory size in bytes. -- `rtcd_plugin_calls_process_virtual_memory_bytes`: Virtual memory size in bytes. - -**WebRTC Connection** - -- `rtcd_rtc_conn_states_total`: Total number of RTC connection state changes. -- `rtcd_rtc_errors_total`: Total number of RTC errors. -- `rtcd_rtc_rtp_bytes_total`: Total number of sent/received RTP packets in bytes. -- `rtcd_rtc_rtp_packets_total`: Total number of sent/received RTP packets. -- `rtcd_rtc_rtp_tracks_total`: Total number of incoming/outgoing RTP tracks. -- `rtcd_rtc_sessions_total`: Total number of active RTC sessions. -- `rtcd_rtc_rtp_tracks_writes_time_bucket`: Time taken to write to outgoing RTP tracks. - - - `rtcd_rtc_rtp_tracks_writes_time_sum` - - - `rtcd_rtc_rtp_tracks_writes_time_count` - -**WebSocket** - -- `rtcd_ws_connections_total`: Total number of active WebSocket connections. -- `rtcd_ws_messages_total`: Total number of received/sent WebSocket messages. - -#### Configuration - -A sample Prometheus configuration to scrape both plugin and `rtcd` metrics could look like this: - -``` -scrape_configs: -- job_name: node - static_configs: - - targets: ['rtcd-0:9100','rtcd-1:9100', 'calls-offloader-1:9100', 'calls-offloader-2:9100'] -- job_name: calls - metrics_path: /plugins/com.mattermost.calls/metrics - static_configs: - - targets: ['app-0:8067','app-1:8067','app-2:8067'] -- job_name: rtcd - static_configs: - - targets: ['rtcd-0:8045', 'rtcd-1:8045'] -``` - -### System tunings - -If you want to host many calls or calls with a large number of participants, take a look at the following platform specific (Linux) tunings (this is the only officially supported target for the plugin right now): - -``` -# Setting the maximum buffer size of the receiving UDP buffer to 16MB -net.core.rmem_max = 16777216 - -# Setting the maximum buffer size of the sending UDP buffer to 16MB -net.core.wmem_max = 16777216 - -# Allow to allocate more memory as needed for more control messages that need to be sent for each socket connected -net.core.optmem_max = 16777216 -``` - -## The rtcd service +### Kubernetes Deployments ```{include} ../../_static/badges/ent-plus.md ``` -The Calls plugin has a built-in [Selective Forwarding Unit (SFU)](https://bloggeek.me/webrtcglossary/sfu/) to route audio and screensharing data. This is the `integrated` option described in the [Modes of operation](#modes-of-operation) section above. But this SFU functionality can be deployed separately as an external `rtcd` instance. +RTCD is the only officially supported approach for Kubernetes deployments. For detailed information on deploying Mattermost Calls in Kubernetes environments, including Helm chart configurations, resource requirements, and scaling considerations, see the [Calls Deployment on Kubernetes](calls-kubernetes.md) guide. -### Reasons to use the `rtcd` service +## When to Use RTCD -This section will help you understand when and why your organization would want to use `rtcd`. +This section will help you understand when and why your organization would want to use the dedicated RTCD service. ```{note} -`rtcd` is a standalone service, which adds operational complexity, maintenance costs, and requires an enterprise licence. For those who are evaluating Calls, and for many small instances of Mattermost, the integrated SFU (the one included in the Calls plugin) may be sufficient initially. +RTCD is a standalone service, which adds operational complexity, maintenance costs, and requires an Enterprise license. For those who are evaluating Calls, and for many small instances of Mattermost, the integrated SFU (the one included in the Calls plugin) may be sufficient initially. ``` -The `rtcd` service is the recommended way to host Calls for the following reasons: +The RTCD service is the recommended way to host Calls for the following reasons: -- **Performance of the main Mattermost server(s).** When the Calls plugin runs the SFU, calls traffic is added to the processing load of the server running the rest of your Mattermost services. If Calls traffic spikes, it can negatively affect the responsiveness of these services. Using an rtcd service isolates the calls traffic processing to those rtcd instances, and also reduces costs by minimizing CPU usage spikes. +- **Performance of the main Mattermost server(s)**: When the Calls plugin runs the SFU, calls traffic is added to the processing load of the server running the rest of your Mattermost services. If Calls traffic spikes, it can negatively affect the responsiveness of these services. Using an RTCD service isolates the calls traffic processing to those RTCD instances, and also reduces costs by minimizing CPU usage spikes. -- **Performance, scalability, and stability of the Calls product.** If Calls traffic spikes, or more overall capacity is needed, `rtcd` servers can be added to balance the load. As an added benefit, if the Mattermost traffic spikes, or if a Mattermost instance needs to be restarted, those people in a current call will not be affected - current calls won't be dropped. +- **Performance, scalability, and stability of the Calls product**: If Calls traffic spikes, or more overall capacity is needed, RTCD servers can be added to balance the load. As an added benefit, if the Mattermost traffic spikes, or if a Mattermost instance needs to be restarted, those people in a current call will not be affected - current calls won't be dropped. -Some caveats apply here. Web socket events (for example: emoji reactions, hand raising, muting/unmuting) will not be transmitted while the main Mattermost server is down. But the call itself will continue while the main server restarts. + Some caveats apply here. While the main Mattermost server is down, things will quickly start to go awry: WebSocket events (for example: emoji reactions, hand raising, muting/unmuting) will not be transmitted, and new media track negotiations will fail. While existing media streams will continue initially, if the main server does not come back up within a reasonable timeframe (e.g., 1 minute), clients will begin to drop from the call entirely as their WebSocket re-connect attempts will hit the maximum allowed. -- **Kubernetes deployments.** In a Kubernetes deployment, `rtcd` is strongly recommended; it is currently the only officially supported way to run Calls. -- **Technical benefits.** The dedicated `rtcd` service has been optimized and tuned at the system/network level for real-time audio/video traffic, where latency is generally more important than throughput. +- **Kubernetes deployments**: In a Kubernetes deployment, RTCD is strongly recommended; it is currently the only officially supported way to run Calls. -In general, `rtcd` is the preferred solution for a performant and scalable deployment. With `rtcd`, the Mattermost server will be minimally impacted when hosting a high number of calls. +- **Technical benefits**: The dedicated RTCD service has been optimized and tuned at the system/network level for real-time audio/video traffic, where latency is generally more important than throughput. -See the [Mattermost rtcd repository documentation](https://github.com/mattermost/rtcd/blob/master/README.md) on GitHub for details on [how to run calls through the service](https://github.com/mattermost/rtcd/blob/master/docs/getting_started.md), as well as: +In general, RTCD is the preferred solution for a performant and scalable deployment. With RTCD, the Mattermost server will be minimally impacted when hosting a high number of calls. -- [Key implementation details](https://github.com/mattermost/rtcd/blob/master/docs/implementation.md) -- [Project structure](https://github.com/mattermost/rtcd/blob/master/docs/project_structure.md) -- [Configuration overrides](https://github.com/mattermost/rtcd/blob/master/docs/env_config.md) -- [Authentication flow](https://github.com/mattermost/rtcd/blob/master/docs/security.md) +For detailed RTCD setup instructions, see the [RTCD Setup and Configuration](calls-rtcd-setup.md) guide. -### Horizontal scalability +## Call Recording and Transcription ```{include} ../../_static/badges/ent-plus.md ``` -The supported way to enable horizontal scalability for Calls is through a form of DNS based load balancing. This can be achieved regardless of how the `rtcd` service is deployed (bare bone instance, Kubernetes, or an alternate way). - -In order for this to work, the [RTCD Service URL](https://docs.mattermost.com/configure/plugins-configuration-settings.html#rtcd-service-url) should point to a hostname that resolves to multiple IP addresses, each pointing to a running `rtcd` instance. The Mattermost Calls plugin will then automatically distribute calls amongst the available hosts. +Mattermost Calls supports recording, transcription, and live captioning features. These processes are resource-intensive and MUST be offloaded from the Mattermost application server to maintain performance. -The expected requirements are the following: +For call recording and transcription, you need to: -- When a new `rtcd` instance is deployed, it should be added to the DNS record. The plugin side will then be able to pick it up and start assigning calls to the new host. -- If a `rtcd` instance goes down, it should be removed from the DNS record. The plugin side can then detect the change and stop assigning new calls to that host. +1. Deploy the `calls-offloader` service +2. Configure the service URL in the System Console +3. Enable call recordings and/or transcriptions in the plugin settings -```{note} -- Load balancing is done at the call level. This means that a single call will always live on a single `rtcd` instance. -- There's currently no support for spreading sessions belonging to the same call across a fleet of instances. -``` +For detailed setup instructions, see the [Calls Offloader Setup and Configuration](calls-offloader-setup.md) guide. -## Configure recording, transcriptions, and live captions +## Air-Gapped Deployments -```{include} ../../_static/badges/ent-plus.md -``` - -Before you can start recording, transcribing, and live captioning calls, you need to configure the `calls-offloader` job service. See the [calls-offloader](https://github.com/mattermost/calls-offloader/blob/master/docs/getting_started.md) documentation on GitHub for details on deploying and running this service. [Performance and scalability recommendations](https://github.com/mattermost/calls-offloader/blob/master/docs/performance.md) related to this service are also available on GitHub. - -```{note} -If deploying the service in a Kubernetes cluster, refer to the later section on [Helm charts](#helm-charts). -``` - -Once the `calls-offloader` service is running, recordings should be explicitly enabled through the [Enable call recordings](https://docs.mattermost.com/configure/plugins-configuration-settings.html#enable-call-recordings) config setting and the service's URL should be configured using [Job service URL](https://docs.mattermost.com/configure/plugins-configuration-settings.html#job-service-url). - -Call transcriptions can be enabled through the [Enable call transcriptions](https://docs.mattermost.com/configure/plugins-configuration-settings.html#enable-call-transcriptions) configuration setting. +Mattermost Calls can function in air-gapped environments. Exposing Calls to the public internet is only necessary when users need to connect from outside the local network, and no existing method supports that connection. In such setups: -Live captions can be enabled through the [Enable live captions](https://docs.mattermost.com/configure/plugins-configuration-settings.html#enable-live-captions) configuration setting. +- Users should connect from within the private/local network. This can be done on-premises, through a VPN, or via virtual machines. +- Configuring a STUN server is unnecessary, as all connections occur within the local network. +- The [ICE Host Override](https://docs.mattermost.com/configure/plugins-configuration-settings.html#ice-host-override) configuration setting can be optionally set with a local IP address (e.g., 192.168.1.45), depending on the specific network configuration and topology. +- For call recording and transcription in air-gapped environments, see the [Air-Gapped Deployments](calls-offloader-setup.md#air-gapped-deployments) section in the Calls Offloader Setup documentation. ```{note} -- The call transcriptions functionality is available starting in Calls version v0.22.0. -- The live captions functionality is available starting in Calls version v0.26.2. -``` - -## Kubernetes deployments - -```{include} ../../_static/badges/ent-plus.md +For RHEL deployments with firewalld or fapolicyd, see the [RHEL deployments with firewalld and fapolicyd](#rhel-deployments-with-firewalld-and-fapolicyd) section above. ``` -The Calls plugin has been designed to integrate well with Kubernetes to offer improved scalability and control over the deployment. - -This is a sample diagram showing how the `rtcd` standalone service can be deployed in a Kubernetes cluster: - -![A diagram of calls deployed in a Kubernetes cluster.](/images/calls-deployment-kubernetes.png) - -If Mattermost isn't deployed in a Kubernetes cluster, and you want to use this deployment type, see the [Deploy Mattermost on Kubernetes](https://docs.mattermost.com/install/install-kubernetes.html) documentation. - -### Helm Charts - -The recommended way to deploy Calls related components and services in a Kubernetes deployment is to use the officially provided Helm charts. Related documentation including detailed information on how to deploy these services can be found in our `mattermost-helm` repository: +## Performance Considerations -- [rtcd Helm chart](https://github.com/mattermost/mattermost-helm/tree/master/charts/mattermost-rtcd) +Calls performance primarily depends on: -- [calls-offloader Helm chart](https://github.com/mattermost/mattermost-helm/tree/master/charts/mattermost-calls-offloader) +- **CPU resources**: More participants require more processing power +- **Network bandwidth**: Both incoming and outgoing traffic increases with participant count. Due to the nature of the service, the bottleneck is always going to be the outgoing/egress path +- **Active speakers**: Unmuted participants require significantly more resources +- **Presenters**: Screen sharing participants require even more resources than active speakers -### Limitations - -Due to the inherent complexities of hosting a WebRTC service, some limitations apply when deploying Calls in a Kubernetes environment. - -One key requirement is that each `rtcd` process live in a dedicated Kubernetes node. This is necessary to forward the data correctly while allowing for horizontal scaling. Data should generally not go through a standard ingress but directly to the pod running the `rtcd` process. - -The general recommendation is to expose one external IP address per `rtcd` instance (Kubernetes node). This makes it simpler to scale as the application is able to detect its own external address (through STUN) and advertise it to clients to achieve connectivity with minimal configuration. - -If, for some reason, exposing multiple IP addresses is not possible in your environment, port mapping (NAT) can be used. In this scenario different ports are used to map the respective `rtcd` nodes behind the single external IP. Example: - -```sh -EXT_IP:8443 -> rtcdA:8443 -EXT_IP:8444 -> rtcdB:8443 -EXT_IP:8445 -> rtcdC:8443 -``` +For detailed performance metrics, benchmarks, and monitoring guidance, see the [Calls Metrics and Monitoring](calls-metrics-monitoring.md) guide. -This case requires a couple of extra configurations: - -- NAT mappings need to be in place for every `rtcd` node. This is usually done at the ingress point (e.g., ELB, NLB, etc). -- The `RTCD_RTC_ICEHOSTPORTOVERRIDE` config should be used to pass a full mapping of node IPs and their respective port. - - Example: `RTCD_RTC_ICEHOSTPORTOVERRIDE=rtcdA_IP/8443,rtcdB_IP/8444,rtcdC_IP/8445` -- The `RTCD_RTC_ICEHOSTOVERRIDE` should be used to set the external IP address. - -```{note} -One option to limit these static mappings is to reduce the size of the local subnet (e.g., to `/29`). -``` - -## Frequently asked questions - -### Is there encryption? +## Frequently Asked Questions +### Is calls traffic encrypted? Media (audio/video) is encrypted using security standards as part of WebRTC. It's mainly a combination of DTLS and SRTP. It's not e2e encrypted in the sense that in the current design all media needs to go through Mattermost which acts as a media router and has complete access to it. Media is then encrypted back to the clients so it's secured during transit. In short: only the participant clients and the Mattermost server have access to unencrypted call data. ### Are there any third-party services involved? - -The only external service used is a Mattermost official STUN server (`stun.global.calls.mattermost.com`) which is configured as default. This is primarily used to find the public address of the Mattermost instance if none is provided through the [ICE Host Override](https://docs.mattermost.com/configure/plugins-configuration-settings.html#ice-host-override) option. The only information sent to this service is the IP addresses of clients connecting as no other traffic goes through it. It can be removed in cases where the [ICE Host Override](https://docs.mattermost.com/configure/plugins-configuration-settings.html#ice-host-override) setting is provided. - -```{note} -In air-gapped deployments, using STUN servers is not necessary since all connections remain within the local network. -``` +Only a Mattermost STUN server (`stun.global.calls.mattermost.com`) is used by default. No user information, call metadata, or media traffic is ever sent to or shared with this STUN service; its sole purpose is to help WebRTC clients discover their public IP address and port mapping. Furthermore, this dependency can be entirely removed if you manually configure the ICE Host Override setting. ### Is using UDP a requirement? +UDP is the recommended protocol to serve real-time media as it allows for the lowest latency between peers, but TCP fallback is supported since plugin version 0.17 and RTCD version 0.11. -Yes, UDP is the recommended protocol to serve real-time media as it allows for the lowest latency between peers. However, there are a couple of possible solutions to cover clients that due to limitations or strict firewalls are unable to use UDP: +If clients are unable to connect using UDP (due to limitations or strict firewalls), you have a few options: - Since plugin version 0.17 and `rtcd` version 0.11 the RTC service will listen for TCP connections in addition to UDP ones. If configured correctly (e.g. using commonly allowed ports such as 80 or 443) it's possible to have clients connect directly through TCP when unable to do it through the preferred UDP channel. - Run calls through an external TURN server that listens on TCP and relays all media traffic between peers. However, this is a sub-optimal solution that should be avoided if possible as it will introduce extra latency along with added infrastructural cost. ### Do I need a TURN server? +Only if clients are behind restrictive firewalls that block UDP. We recommend (and officially support) [coturn](https://github.com/coturn/coturn) if needed. -TURN becomes necessary when you expect to have clients that are unable to connect through the configured UDP port. This can happen due to very restrictive firewalls that either block non standard ports even in the outgoing direction or don't allow the use of the UDP protocol altogether (e.g. some corporate firewalls). In such cases TURN is needed to allow connectivity. - -We officially support and recommend using [coturn](https://github.com/coturn/coturn) for a stable and performant TURN service implementation. - -### How will this work with an existing reverse proxy sitting in front of Mattermost? +### Can the traffic between Mattermost and `rtcd` be kept internal or should it be opened to the public? -Generally clients should connect directly to either Mattermost or, if deployed, the dedicated `rtcd` service through the configured UDP port . However, it's also possible to route the traffic through an existing load balancer as long as this has support for routing the UDP protocol (e.g. nginx). Of course this will require additional configuration and potential changes to how the plugin is run as it won't be possible to load balance the UDP flow across multiple instances like it happens for HTTP. - -### Do calls require a dedicated server to work or can they run alongside Mattermost? - -The plugin can function in different modes. By default calls are handled completely by the plugin which runs as part of Mattermost. It's also possible to use a dedicated service to offload the computational and bandwidth costs and scale further (Enterprise only). - -### Can the traffic between Mattermost and `rtcd` be kept internal or should it be opened to the public? - -When possible, it's recommended to keep communication between the Mattermost cluster and the dedicated `rtcd` service under the same private network as this can greatly simplify deployment and security. There's no requirement to expose `rtcd`'s HTTP API to the public internet. +Yes, the `rtcd` <-> Mattermost communication (HTTP/WebSocket API over TCP port 8045) should remain internal in almost all cases. When possible, it's highly recommended to keep communication between the Mattermost cluster and the dedicated `rtcd` service under the same private network as this minimizes latency and greatly simplifies deployment and security. There is no requirement to expose `rtcd`'s HTTP API to the public internet. Only the media ports (UDP/TCP 8443) need to be accessible to end-users. ### Can Calls be rolled out on a per-channel basis? Yes. Mattermost system admins running self-hosted deployments can enable or disable call functionality per channel. Once [test mode](https://docs.mattermost.com/configure/plugins-configuration-settings.html#test-mode) is enabled for Mattermost Calls: -- Select **Enable calls** for each channel where you want Calls enabled -- Select **Disable calls** for all channels where you want Calls disabled. +1. **Navigate to the channel** where you want to enable or disable Calls +2. **Access the channel menu** by clicking the channel name at the top of the channel +3. **Select the Calls option** from the dropdown menu: + - Select **Enable calls** for each channel where you want Calls enabled + - Select **Disable calls** for all channels where you want Calls disabled + +![Channel menu showing Enable/Disable calls options](../../images/calls-channel-enable-disable.png) Once Calls is enabled for specific channels, users can start making calls in those channels. @@ -778,48 +324,11 @@ When [test mode](https://docs.mattermost.com/configure/plugins-configuration-set ## Troubleshooting -### Connectivity issues - -If calls are failing to connect or timing out, it's likely there could be a misconfiguration at either the plugin config or networking level. - -For example, the [RTC Server Port (UDP)](https://docs.mattermost.com/configure/plugins-configuration-settings.html#rtc-server-port-udp) or the [RTC Server Port (TCP)](https://docs.mattermost.com/configure/plugins-configuration-settings.html#rtc-server-port-tcp) may not be open or forwarded correctly. - -#### Connectivity checks - -An easy way to check whether data can go through is to perform some tests using the `netcat` command line tool. - -On the host running Calls (could be the Mattermost instance itself or the one running `rtcd` depending on the chosen setup), run the following: - -```sh -nc -l -u -p 8443 -``` - -On the client side (i.e., the machine you would normally use to run the Mattermost desktop app or browser), run the following: +For comprehensive troubleshooting steps and debugging techniques, please refer to the [Calls Troubleshooting](calls-troubleshooting.md) guide. -```sh -nc -v -u HOST_IP 8443 -``` - -If connection succeeds, you should be able to send and receive text messages by typing and hitting enter on either side. - -```{note} -`HOST_IP` should generally be the public (client facing) IP of the Mattermost -(or `rtcd`) instance hosting the calls. When set, it should be the value of the [ICE Host Override](https://docs.mattermost.com/configure/plugins-configuration-settings.html#ice-host-override) -config setting. - -`8443` should be changed with the port configured in [RTC Server Port](https://docs.mattermost.com/configure/plugins-configuration-settings.html#rtc-server-port-udp). - -The same checks can be performed to test connectivity through the TCP port using the same commands with `-u` flag removed. -``` - -#### Network packets debugging - -A more advanced way to debug networking issues is to use the `tcpdump` command line utility to temporaily monitor network packets flowing in and out of the instance hosting calls. - -On the server side, run the following: - -```sh -sudo tcpdump -n port 8443 -``` +## Next Steps -This command will output information (i.e. source and destination addresses) for all the network packets being sent or received through port `8443`. This is a good way to check whether data is getting in and out of the instance and can be used to quickly identify network configuration issues. +1. For detailed setup instructions, see [RTCD Setup and Configuration](calls-rtcd-setup.md) +2. For monitoring guidance, see [Calls Metrics and Monitoring](calls-metrics-monitoring.md) +3. If you encounter issues, see [Calls Troubleshooting](calls-troubleshooting.md) +4. For Kubernetes deployments, see [Calls Deployment on Kubernetes](calls-kubernetes.md) diff --git a/source/administration-guide/configure/calls-kubernetes.md b/source/administration-guide/configure/calls-kubernetes.md new file mode 100644 index 00000000000..51f413fa4a8 --- /dev/null +++ b/source/administration-guide/configure/calls-kubernetes.md @@ -0,0 +1,110 @@ +# Calls Deployment on Kubernetes + +```{include} ../../_static/badges/all-commercial.md +``` + +This guide provides detailed information for deploying Mattermost Calls on Kubernetes environments. + +## Overview + +Mattermost Calls has been designed to integrate well with Kubernetes to offer improved scalability and control over the deployment. For Kubernetes deployments, the RTCD service is strongly recommended and is the only officially supported approach. + +## Architecture + +![Calls deployed in a Kubernetes cluster](../../images/calls-deployment-kubernetes.png) + +This diagram shows how the RTCD standalone service can be deployed in a Kubernetes cluster. In this architecture: + +1. Calls traffic is handled by dedicated RTCD pods +2. Scaling is managed through Kubernetes deployment configurations +3. Call recording and transcription is handled by the calls-offloader service (see [Calls Offloader Setup and Configuration](calls-offloader-setup.md)) + +If Mattermost isn't already deployed in your Kubernetes cluster and you want to use this deployment type, visit the [Kubernetes operator guide](/install/mattermost-kubernetes-operator.md). + +## Helm Chart Deployment + +The recommended way to deploy Calls-related components in a Kubernetes environment is to use the officially provided Helm charts: + +### RTCD Helm Chart + +The RTCD Helm chart deploys the RTCD service needed for call media handling: + +```bash +helm repo add mattermost https://helm.mattermost.com +helm repo update + +helm install mattermost-rtcd mattermost/mattermost-rtcd \ + --set service.annotations."service\\.beta\\.kubernetes\\.io/aws-load-balancer-backend-protocol"=udp +``` + +For complete configuration options, see the [RTCD Helm chart documentation](https://github.com/mattermost/mattermost-helm/tree/master/charts/mattermost-rtcd). + +### Calls-Offloader Helm Chart + +If you need call recording and transcription capabilities, deploy the calls-offloader service: + +```bash +helm install mattermost-calls-offloader mattermost/mattermost-calls-offloader \ + --set ingress.enabled=true \ + --set ingress.host=calls-offloader.example.com +``` + +For complete configuration options, see the [Calls-Offloader Helm chart documentation](https://github.com/mattermost/mattermost-helm/tree/master/charts/mattermost-calls-offloader). + +## Kubernetes-Specific Configuration + +### Network Configuration + +For Kubernetes deployments, you need to ensure specific connectivity paths: + +1. **Client to RTCD connectivity**: UDP and TCP traffic on port 8443 is properly routed from clients to RTCD pods (for media, with TCP acting as a fallback). +2. **Mattermost to RTCD API connectivity**: There needs to be a clear connectivity path between Mattermost and RTCD on the API port (TCP 8045) +3. **Network policies**: Network policies must allow the required communications between Mattermost and RTCD services + +### Resource Requirements + +Resource requirements for RTCD pods depend heavily on the expected call volume, participant count, and whether screen sharing is used. + +We strongly recommend reviewing the [Performance Baselines](calls-metrics-monitoring.md#performance-baselines) to determine the appropriate CPU, memory, and network requests and limits for your specific deployment needs rather than relying on generic defaults. + +### Scaling Considerations + +Horizontal scaling of RTCD pods is possible, but remember: + +1. Each call is hosted entirely on a single RTCD pod +2. DNS A records should be set up for all RTCD pods so server can discover RTCD servers. +3. Health checks should ensure that only healthy pods are reported via DNS +4. Calls remain on their assigned pod for their entire duration + +### Limitations + +Due to the inherent complexities of hosting a WebRTC service, some limitations apply when deploying Calls in a Kubernetes environment. + +One key requirement is that each `rtcd` process must live in a dedicated Kubernetes node. This is necessary to forward the data correctly while allowing for horizontal scaling. Data should generally direct directly to the pod running the `rtcd` process, as routing this traffic through a standard ingress is not recommended for RTCD deployments. + +The general recommendation is to expose one external IP address per `rtcd` instance (Kubernetes node). This makes it simpler to scale as the application is able to detect its own external address (through STUN) and advertise it to clients to achieve connectivity with minimal configuration. + + + +## Monitoring and Metrics + +For detailed information on metrics collection and monitoring, see the [Calls Metrics and Monitoring](calls-metrics-monitoring.md) guide. + +## Troubleshooting + +For Kubernetes-specific troubleshooting: + +1. Check pod logs: `kubectl logs -f deployment/mattermost-rtcd` +2. Verify service connectivity: `kubectl port-forward service/mattermost-rtcd 8045:8045` +3. Ensure UDP and TCP traffic is properly routed through your load balancer +4. Verify network policies allow required communication paths + +For detailed troubleshooting steps, see the [Calls Troubleshooting](calls-troubleshooting.md) guide. + +## Other Calls Documentation + +- [Calls Overview](calls-deployment.md): Overview of deployment options and architecture +- [RTCD Setup and Configuration](calls-rtcd-setup.md): Comprehensive guide for setting up the dedicated RTCD service +- [Calls Offloader Setup and Configuration](calls-offloader-setup.md): Setup guide for call recording and transcription +- [Calls Metrics and Monitoring](calls-metrics-monitoring.md): Guide to monitoring Calls performance using metrics and observability +- [Calls Troubleshooting](calls-troubleshooting.md): Detailed troubleshooting steps and debugging techniques diff --git a/source/administration-guide/configure/calls-metrics-monitoring.md b/source/administration-guide/configure/calls-metrics-monitoring.md new file mode 100644 index 00000000000..d5ae641d3db --- /dev/null +++ b/source/administration-guide/configure/calls-metrics-monitoring.md @@ -0,0 +1,418 @@ +# Calls Metrics and Monitoring + +```{include} ../../_static/badges/ent-plus.md +``` + +This guide provides detailed information on monitoring Mattermost Calls performance and health through metrics and observability tools. Effective monitoring is essential for maintaining optimal call quality and quickly addressing any issues that arise. + +- [Metrics overview](#metrics-overview) +- [Setting up monitoring](#setting-up-monitoring) +- [Key metrics to monitor](#key-metrics-to-monitor) +- [Performance baselines](#performance-baselines) +- [Troubleshooting metrics collection](#troubleshooting-metrics-collection) + +## Metrics Overview + +Mattermost Calls provides metrics through Prometheus for both the Calls plugin and the RTCD service. These metrics help track: + +- Active call sessions and participants +- Media track statistics +- Connection states and errors +- Resource utilization (CPU, memory, network) +- WebSocket connections and events + +The metrics are exposed through HTTP endpoints: + +- **Calls Plugin**: `/plugins/com.mattermost.calls/metrics` +- **RTCD Service**: `/metrics` (default) or a configured endpoint + +Resource utilization metrics (CPU, memory, network) are mainly provided by an external service ([`node-exporter`](https://prometheus.io/docs/guides/node-exporter/)). + +> Metrics for the calls plugin are exposed through the `/plugins/com.mattermost.calls/metrics` subpath under the existing Mattermost server metrics endpoint. This is controlled by the [Listen address for performance](https://docs.mattermost.com/configure/environment-configuration-settings.html#listen-address-for-performance) configuration setting. It defaults to port 8067. For example: `http://localhost:8067/plugins/com.mattermost.calls/metrics` +> The RTCD Service `/metrics` endpoint is exposed on the HTTP API (e.g. `http://localhost:8045/metrics`). + +## Setting Up Monitoring + +For instructions on deploying Prometheus and Grafana for Mattermost, please refer to the [Deploy Prometheus and Grafana for Performance Monitoring](https://docs.mattermost.com/scale/deploy-prometheus-grafana-for-performance-monitoring.html) guide. + +Once Prometheus and Grafana are set up, you will need to configure Prometheus to scrape metrics from the Calls-related services. + +### Prometheus Scrape Configuration + +Add the following jobs to your `prometheus.yml` configuration: + +```yaml +scrape_configs: + - job_name: 'calls-plugin' + metrics_path: /plugins/com.mattermost.calls/metrics + static_configs: + - targets: ['MATTERMOST_SERVER_IP:8067'] + labels: + service_name: 'calls-plugin' + + - job_name: 'rtcd' + metrics_path: /metrics + static_configs: + - targets: ['RTCD_SERVER_IP:8045'] + labels: + service_name: 'rtcd' + + - job_name: 'rtcd-node-exporter' + metrics_path: /metrics + static_configs: + - targets: ['RTCD_SERVER_IP:9100'] + labels: + service_name: 'rtcd' + + - job_name: 'calls-offloader-node-exporter' + metrics_path: /metrics + static_configs: + - targets: ['CALLS_OFFLOADER_SERVER_IP:9100'] + labels: + service_name: 'offloader' +``` + +Replace the placeholder IP addresses with your actual server addresses: + +- `MATTERMOST_SERVER_IP`: IP address of your Mattermost server +- `RTCD_SERVER_IP`: IP address of your RTCD server +- `CALLS_OFFLOADER_SERVER_IP`: IP address of your calls-offloader server (if deployed) + +```{important} +**Metrics Configuration Notice**: Use the `service_name` labels as shown in the configuration above. These labels help organize metrics in dashboards and enable proper service identification. +``` + +```{note} +- **node_exporter**: Optional but recommended for system-level metrics (CPU, memory, disk, network). See [node_exporter setup guide](https://prometheus.io/docs/guides/node-exporter/) for installation instructions. +- **calls-offloader**: Only needed if you have call recording/transcription enabled. +``` + +### Mattermost Calls Grafana Dashboard + +You can use the official [Mattermost Calls Performance Monitoring](https://grafana.com/grafana/dashboards/23225-mattermost-calls-performance-monitoring/) dashboard to visualize these metrics. + +- To import it directly into Grafana, use dashboard ID: `23225`. +- The dashboard is also available as JSON source from the [Mattermost performance assets repository](https://github.com/mattermost/mattermost-performance-assets/blob/master/grafana/mattermost-calls-performance-monitoring.json) for manual import or customization. + +## Key Metrics to Monitor + +### RTCD Metrics + +#### Process Metrics + +These metrics help monitor the health and resource usage of the RTCD process: + +- `rtcd_process_cpu_seconds_total`: Total CPU time spent +- `rtcd_process_open_fds`: Number of open file descriptors +- `rtcd_process_max_fds`: Maximum number of file descriptors +- `rtcd_process_resident_memory_bytes`: Memory usage in bytes +- `rtcd_process_virtual_memory_bytes`: Virtual memory used + + + +#### WebRTC Connection Metrics + +These metrics track the WebRTC connections and media flow: + +- `rtcd_rtc_conn_states_total{state="X"}`: Count of connections in different states +- `rtcd_rtc_errors_total{type="X"}`: Count of RTC errors by type +- `rtcd_rtc_rtp_tracks_total{direction="X"}`: Count of RTP tracks (incoming/outgoing) +- `rtcd_rtc_sessions_total`: Total number of active RTC sessions + + + +#### WebSocket Metrics + +These metrics track the signaling channel: + +- `rtcd_ws_connections_total`: Total number of active WebSocket connections. This is about RTCD <-> MM, so the connection count should match the number of MM nodes. +- `rtcd_ws_messages_total{direction="X"}`: Count of WebSocket messages (sent/received) + + + +### Calls Plugin Metrics + +Similar metrics are available for the Calls plugin with the following prefixes: + +- Process metrics: `mattermost_plugin_calls_process_*` +- WebRTC connection metrics: `mattermost_plugin_calls_rtc_*` +- WebSocket metrics: `mattermost_plugin_calls_websocket_*` +- Store metrics: `mattermost_plugin_calls_store_ops_total` + +## Performance Baselines + +The following performance benchmarks provide baseline metrics for RTCD deployments under various load conditions and configurations. + +**Deployment specifications** + +- 1x r6i.large nginx proxy +- 3x c5.large MM app nodes (HA) +- 2x db.x2g.xlarge RDS Aurora MySQL v8 (one writer, one reader) +- 1x (c7i.xlarge, c7i.2xlarge, c7i.4xlarge) RTCD +- 2x c7i.2xlarge load-test agents + +**App specifications** + +- Mattermost v9.6 +- Mattermost Calls v0.28.0 +- RTCD v0.16.0 +- load-test agent v0.28.0 + +**Media specifications** + +- Speech sample bitrate: 80Kbps +- Screen sharing sample bitrate: 1.6Mbps + +**Results** + +Below are the detailed benchmarks based on internal performance testing: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CallsParticipants/callUnmuted/callScreen sharingCPU (avg)Memory (avg)Bandwidth (in/out)Instance type (RTCD)
110002no47%1.46GB1Mbps / 194Mbpsc7i.xlarge
18001yes64%1.43GB2.7Mbps / 1.36Gbpsc7i.xlarge
110001yes79%1.54GB2.9Mbps / 1.68Gbpsc7i.xlarge
101001yes74%1.56GB18.2Mbps / 1.68Gbpsc7i.xlarge
100102no49%1.46GB18.7Mbps / 175Mbpsc7i.xlarge
100101yes84%1.73GB171Mbps / 1.53Gbpsc7i.xlarge
110002no20%1.44GB1.4Mbps / 194Mbpsc7i.2xlarge
110002yes49%1.53GB3.6Mbps / 1.79Gbpsc7i.2xlarge
210001yes73%2.38GB5.7Mbps / 3.06Gbpsc7i.2xlarge
100102yes60%1.74GB181Mbps / 1.62Gbpsc7i.2xlarge
150101yes72%2.26GB257Mbps / 2.30Gbpsc7i.2xlarge
150102yes79%2.34GB271Mbps / 2.41Gbpsc7i.2xlarge
250102no58%2.66GB47Mbps / 439Mbpsc7i.2xlarge
100022no78%2.31GB178Mbps / 195Mbpsc7i.2xlarge
210002yes41%2.6GB7.23Mbps / 3.60Gbpsc7i.4xlarge
310002yes63%3.53GB10.9Mbps / 5.38Gbpsc7i.4xlarge
410002yes83%4.40GB14.5Mbps / 7.17Gbpsc7i.4xlarge
250102yes79%3.49GB431Mbps / 3.73Gbpsc7i.4xlarge
50022yes71%2.54GB896Mbps / 919Mbpsc7i.4xlarge
+ +## Troubleshooting Metrics Collection + +### Verify RTCD Metrics are Being Collected + +To verify that Prometheus is successfully collecting RTCD metrics, use this command: + +```bash +curl http://PROMETHEUS_IP:9090/api/v1/label/__name__/values | jq '.' | grep rtcd +``` + +This command queries Prometheus for all available metric names and filters for RTCD-related metrics. + +If no RTCD metrics appear, check: +1. RTCD is running +2. Prometheus is configured to scrape the RTCD metrics endpoint +3. RTCD metrics port is accessible from Prometheus (default: 8045) + +### Check Prometheus Scrape Targets + +To verify all Calls-related services are being scraped successfully: + +1. Open the Prometheus web interface (typically `http://PROMETHEUS_IP:9090`) +2. Navigate to **Status > Targets** +3. Look for your configured Calls services: + - Mattermost server (for Calls plugin metrics) + - RTCD service + +Each target should show status "UP" in green. If a target shows "DOWN" or errors: +- Verify the service is running +- Check network connectivity between Prometheus and the target +- Verify the metrics endpoint is accessible + +## Other Calls Documentation + +- [Calls Overview](calls-deployment.md): Overview of deployment options and architecture +- [RTCD Setup and Configuration](calls-rtcd-setup.md): Comprehensive guide for setting up the dedicated RTCD service +- [Calls Offloader Setup and Configuration](calls-offloader-setup.md): Setup guide for call recording and transcription +- [Calls Deployment on Kubernetes](calls-kubernetes.md): Detailed guide for deploying Calls in Kubernetes environments +- [Calls Troubleshooting](calls-troubleshooting.md): Detailed troubleshooting steps and debugging techniques + +**Note:** +Configure Prometheus storage accordingly to balance disk usage with retention needs. If you need to be tight on storage, you can use a short retention period. If you have lots of storage you can keep the retention length longer. \ No newline at end of file diff --git a/source/administration-guide/configure/calls-offloader-setup.md b/source/administration-guide/configure/calls-offloader-setup.md new file mode 100644 index 00000000000..a3e32e6bf87 --- /dev/null +++ b/source/administration-guide/configure/calls-offloader-setup.md @@ -0,0 +1,501 @@ +# Calls Offloader Setup and Configuration + +```{include} ../../_static/badges/ent-plus.md +``` + + +This guide provides detailed instructions for setting up, configuring, and validating the Mattermost calls-offloader service used for call recording and transcription features. + +- [Overview](#overview) +- [Prerequisites](#prerequisites) +- [Installation and deployment](#installation-and-deployment) +- [Configuration](#configuration) +- [Validation and testing](#validation-and-testing) +- [Integration with Mattermost](#integration-with-mattermost) +- [Troubleshooting](#troubleshooting) +- [Air-Gapped Deployments](#air-gapped-deployments) + +## Overview + +The calls-offloader service is a dedicated microservice that handles resource-intensive tasks for Mattermost Calls, including: + +- **Call recording**: Captures audio and screen sharing content from calls +- **Call transcription**: Provides automated transcription of recorded calls +- **Live captions** (Experimental): Real-time transcription during active calls + +By offloading these tasks to a dedicated service, the main Mattermost server and RTCD service can focus on core functionality while maintaining optimal performance. + +## Prerequisites + +Before deploying calls-offloader, ensure you have: + +- A Mattermost Enterprise license +- A properly configured Mattermost Calls deployment (either integrated or with RTCD) +- A moderately powerful server with Docker installed and running +- Sufficient storage space for recordings (see [Storage Requirements](#storage-requirements)) + +### System Requirements + +For detailed system requirements and performance recommendations, refer to the [calls-offloader performance documentation](https://github.com/mattermost/calls-offloader/blob/master/docs/performance.md). + +### Storage Requirements + +Call recordings can consume significant storage space. Based on average recording sizes with screen sharing on (including one audio track), storage usage by quality chosen is approximately: + +- **Low**: ~0.5GB/hour (or ~8MB/minute) +- **Medium**: ~0.7GB/hour (or ~12MB/minute) +- **High**: ~1.2GB/hour (or ~20MB/minute) + +*Note: Audio-only recordings consume approximately 1MB per minute per participant.* + +## Installation and Deployment + +### Bare Metal or VM Deployment + +```{tip} +Looking for an automated setup? Check out these community-maintained [Calls Installation Scripts](https://github.com/bgardner8008/calls-install-scripts) for quick provisioning of the Calls Offloader service on Ubuntu/Debian systems. +``` + +1. Download the latest release from the [calls-offloader GitHub repository](https://github.com/mattermost/calls-offloader/releases) + +2. Create the necessary directories: + + ```bash + sudo mkdir -p /opt/calls-offloader/data/db + sudo useradd --system --home /opt/calls-offloader calls-offloader + sudo chown -R calls-offloader:calls-offloader /opt/calls-offloader + ``` + +3. Create a configuration file (`/opt/calls-offloader/config.toml`): + + ```toml + [api] + http.listen_address = ":4545" + http.tls.enable = false + http.tls.cert_file = "" + http.tls.cert_key = "" + security.allow_self_registration = true + security.enable_admin = false + security.admin_secret_key = "" + security.session_cache.expiration_minutes = 1440 + + [store] + data_source = "/opt/calls-offloader/data/db" + + [jobs] + api_type = "docker" + max_concurrent_jobs = 100 + failed_jobs_retention_time = "30d" + image_registry = "mattermost" + + [logger] + enable_console = true + console_json = false + console_level = "INFO" + enable_file = true + file_json = true + file_level = "DEBUG" + file_location = "/opt/calls-offloader/calls-offloader.log" + enable_color = true + ``` + +4. Create a systemd service file (`/etc/systemd/system/calls-offloader.service`): + + ```ini + [Unit] + Description=Mattermost Calls Offloader Service + After=network.target docker.service + Requires=docker.service + + [Service] + Type=simple + User=calls-offloader + WorkingDirectory=/opt/calls-offloader + ExecStart=/opt/calls-offloader/calls-offloader --config /opt/calls-offloader/config.toml + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` + +5. Enable and start the service: + + ```bash + sudo systemctl daemon-reload + sudo systemctl enable calls-offloader + sudo systemctl start calls-offloader + ``` + +6. Check the service status: + + ```bash + sudo systemctl status calls-offloader + ``` + +7. Verify the service is responding: + + ```bash + curl http://localhost:4545/version + # Example output: + # {"buildDate":"2025-03-10 19:13","buildVersion":"v0.9.2","buildHash":"a4bd418","goVersion":"go1.23.6"} + ``` + +## Configuration + +### API Configuration + +The API section controls how the service accepts requests: + +- **http.listen_address**: The address and port where the service listens (default: `:4545`) +- **http.tls.enable**: Whether to use TLS encryption for the API +- **security.allow_self_registration**: Allow clients to self-register for job management +- **security.enable_admin**: Enable admin functionality +- **security.admin_secret_key**: Secret key for admin authentication (change from default!) + +### Store Configuration + +Controls persistent data storage: + +- **data_source**: Path to directory for storing client (i.e., connecting Mattermost nodes) IDs and credentials + +### Jobs Configuration + +Controls job processing behavior: + +- **api_type**: Job execution backend (`docker` or `kubernetes`) +- **max_concurrent_jobs**: Maximum number of simultaneous recording/transcription jobs +- **failed_jobs_retention_time**: How long to keep failed job data before cleanup +- **image_registry**: Docker registry for job runner images (typically `mattermost`) + +### Logger Configuration + +Controls logging output: + +- **enable_console**: Log to console output +- **console_json**: Use JSON format for console logs +- **console_level**: Log level for console (DEBUG, INFO, WARN, ERROR) +- **enable_file**: Log to file +- **file_location**: Path to log file +- **enable_color**: Use colored output for console logs + +### Private Network Configuration + +When the Mattermost deployment is running in a private network, additional configuration may be necessary for the jobs spawned by the calls-offloader service to reach the Mattermost server. + +In such cases, you can override the site URL used by recorder jobs or transcriber jobs to connect to Mattermost by setting the following environment variables on the Mattermost server: + +- **MM_CALLS_RECORDER_SITE_URL**: Override the site URL used by recording jobs +- **MM_CALLS_TRANSCRIBER_SITE_URL**: Override the site URL used by transcription jobs + +```{note} +When these Site URL overrides are used, the `ServiceSettings.AllowCorsFrom` setting on your Mattermost server may need to be adjusted accordingly to ensure CORS does not block requests. + +This override configuration lets the recorder and transcriber jobs connect to mattermost server using HTTP instead of HTTPS, which should only be used in a private network. +``` + +Example configuration: + +Create or edit the Mattermost environment file (`/opt/mattermost/config/mattermost.environment`): + +```bash +MM_CALLS_RECORDER_SITE_URL="http://internal-mattermost-server:8065" +MM_CALLS_TRANSCRIBER_SITE_URL="http://internal-mattermost-server:8065" +``` + +Then ensure your Mattermost systemd service references this environment file: + +```ini +[Unit] +Description=Mattermost +After=network.target + +[Service] +Type=notify +EnvironmentFile=/opt/mattermost/config/mattermost.environment +ExecStart=/opt/mattermost/bin/mattermost +TimeoutStartSec=3600 +KillMode=mixed +Restart=always +RestartSec=10 +WorkingDirectory=/opt/mattermost +User=mattermost +Group=mattermost + +[Install] +WantedBy=multi-user.target +``` + +This is particularly useful when: + +- The calls-offloader service runs in a different network segment than clients +- Internal DNS resolution differs from external URLs +- You need to use internal load balancer endpoints for job communication + +## Validation and Testing + +After deploying calls-offloader, validate the installation: + +1. **Check service status**: + + ```bash + # For systemd + sudo systemctl status calls-offloader + ``` + +2. **Test API connectivity**: + + **From the calls-offloader server (localhost test)**: + + ```bash + curl http://localhost:4545/version + # Should return version information + # Example: {"buildDate":"2025-03-10 19:13","buildVersion":"v0.9.2","buildHash":"a4bd418","goVersion":"go1.23.6"} + ``` + + **From the Mattermost server**: + + ```bash + curl http://YOUR_CALLS_OFFLOADER_SERVER:4545/version + # Should return the same version information + # This confirms network connectivity from Mattermost to calls-offloader + ``` + + If the localhost test works but the Mattermost server test fails, check: + + - Firewall rules or SELinux policies on the calls-offloader server (port 4545 must be accessible) + - Network connectivity between Mattermost and calls-offloader servers + +3. **Verify Docker service** (if using docker api_type): + + ```bash + # Check that system user running calls-offloader can access Docker + sudo -u calls-offloader docker ps + ``` + +## Integration with Mattermost + +Once calls-offloader is properly set up and validated, configure Mattermost to use it: + +1. Go to **System Console > Plugins > Calls** + +2. In the **Job Service** section: + + - Set **Job Service URL** to your calls-offloader service (e.g., `http://calls-offloader-server:4545`) + +3. Enable recording and transcription features as needed: + + - **Enable Call Recordings**: Toggle to allow call recordings + - **Enable Call Transcriptions**: Toggle to allow call transcriptions + - **Enable Live Captions** (Experimental): Toggle to allow real-time transcription + +4. Save the configuration + +5. Restart the Calls plugin to re-establish state: + + - Go to **System Console > Plugins > Plugin Management** + - Find the **Calls** plugin and click **Disable** + - Wait a few seconds, then click **Enable** + +6. Test by starting a call and starting a recording + +## Troubleshooting + +### Common Issues + +**"failed to create recording job: max concurrent jobs reached"** + +This error occurs when the calls-offloader service has reached its configured job limit, and it will usually result in a failure message on the Mattermost Calls plugin side (such as a timeout). + +Solutions: + +- Increase `max_concurrent_jobs` in the configuration +- Check if jobs are hanging and restart the service +- Monitor system resources and scale up if needed + +**Jobs not processing** + +Check the following: + +- Verify the calls-offloader service is running: `sudo systemctl status calls-offloader` +- Ensure network connectivity between Mattermost and calls-offloader +- Check Docker daemon is running and accessible by the user running the Calls Offloader service (E.g., user: ``calls-offloader``) +- Verify authentication configuration matches between services +- Review service logs for specific error messages + +**Docker permission issues** + +If using Docker API and seeing permission errors: + +```bash +# Add calls-offloader user to docker group +sudo usermod -a -G docker calls-offloader +sudo systemctl restart calls-offloader +``` + +### Debugging Commands + +Monitor calls-offloader job containers: + +```bash +# View running job containers +docker ps --format "{{.ID}} {{.Image}}" | grep "calls" + +# Follow logs for debugging +docker ps --format "{{.ID}} {{.Image}}" | grep "calls" | awk '{print $1}' | xargs -I {} docker logs -f {} + +# View completed job containers +docker ps -a --filter "status=exited" +``` + +Monitor service health: + +```bash +# Check service version and health +curl http://localhost:4545/version +``` + +Check service logs: + +```bash +# View recent logs +sudo journalctl -u calls-offloader -f + +# View log file (if file logging enabled) +tail -f /opt/calls-offloader/calls-offloader.log +``` + +### Performance Monitoring + +Monitor calls-offloader performance and resource usage to ensure optimal operation. See [Calls Metrics and Monitoring](calls-metrics-monitoring.md) for details on setting up metrics and observability. + +## Air-Gapped Installation of `calls-offloader` + +This guide covers deploying `calls-offloader` in an environment without internet access. The process uses scripts from the [calls-install-scripts](https://github.com/bgardner8008/calls-install-scripts) repository and follows a two-phase workflow: preparing a transfer bundle on an internet-connected machine, then deploying it on the isolated target machine. + +### Overview + +Because `calls-offloader` relies on Docker images for the recorder and transcriber jobs, an air-gapped deployment requires that those images be pre-pulled and packaged alongside the `calls-offloader` binary before being transferred to the target environment. + +The install scripts handle this in two stages: + +1. **Prepare** (internet-connected machine) — `setup-airgap-offloader.sh` pulls the required Docker images and downloads the `calls-offloader` binary, then packages everything into a transfer bundle and generates a ready-to-run deployment script. +2. **Deploy** (air-gapped machine) — Transfer the bundle and run the generated `deploy-airgap-offloader.sh` script, which loads the Docker images into a local registry and installs the service. + +### Prerequisites + +- An internet-connected Linux machine with Docker installed (for the preparation phase) +- A target air-gapped Linux machine with: + - systemd + - Docker installed + - Root or sudo access +- The [calls-install-scripts](https://github.com/bgardner8008/calls-install-scripts) repository cloned on the internet-connected machine + +### Phase 1: Prepare the Transfer Bundle + +On the **internet-connected machine**, run `setup-airgap-offloader.sh` specifying the versions of each component to package: + +```bash +./setup-airgap-offloader.sh \ + --offloader v0.9.5 \ + --recorder v0.9.0 \ + --transcriber v0.3.0 \ + --arch amd64 +``` + +| Flag | Description | Default | +|------|-------------|---------| +| `--offloader VERSION` | `calls-offloader` binary version (e.g. `v0.9.5`) | required | +| `--recorder VERSION` | `calls-recorder` Docker image version | required | +| `--transcriber VERSION` | `calls-transcriber` Docker image version | required | +| `--arch amd64\|arm64` | Target CPU architecture | `amd64` | + +The script will: + +1. Pull the `calls-recorder` and `calls-transcriber` Docker images from Docker Hub +2. Download the `calls-offloader` binary from GitHub releases +3. Save the Docker images as `.tar` archives +4. Generate a `deploy-airgap-offloader.sh` deployment script configured for the selected versions +5. Produce a transfer bundle containing all of the above + +### Phase 2: Transfer to the Air-Gapped Machine + +Copy the generated bundle to the target machine using whatever transfer mechanism is available in your environment (USB drive, secure file transfer, etc.): + +```bash +scp calls-offloader-airgap-bundle.tar.gz user@airgap-host:/tmp/ +``` + +On the air-gapped machine, extract the bundle: + +```bash +tar -xzf calls-offloader-airgap-bundle.tar.gz +cd calls-offloader-airgap-bundle/ +``` + +### Phase 3: Deploy on the Air-Gapped Machine + +Run the generated deployment script with root or sudo privileges: + +```bash +sudo ./deploy-airgap-offloader.sh +``` + +This script will: + +1. Configure the local Docker daemon to use a local image registry +2. Load the packaged Docker images into that registry +3. Install the `calls-offloader` binary to `/usr/local/bin/` +4. Create the `mattermost` system user and add it to the `docker` group +5. Generate and enable a systemd service unit +6. Start the service and verify it is running + +Once complete, verify the service is up: + +```bash +curl http://localhost:4545/version +``` + +### Connecting to Mattermost + +Configure the Mattermost Calls plugin to use the offloader service via **System Console > Plugins > Calls > Job service URL**, setting it to `http://:4545`. + +> [!NOTE] +> The first time Mattermost connects to the offloader it will self-register and store its authentication key in the database, provided `API_SECURITY_ALLOWSELFREGISTRATION=true` is set (the default in the deployment script). + +### Private Network Considerations + +In air-gapped environments the recorder and transcriber containers typically need to reach the Mattermost server via an internal URL. Set the following environment variables on the **Mattermost server** to override the site URL used by spawned jobs: + +``` +MM_CALLS_RECORDER_SITE_URL=http://internal-mattermost-server:8065 +MM_CALLS_TRANSCRIBER_SITE_URL=http://internal-mattermost-server:8065 +``` + +You may also need to add the internal URL to [`ServiceSettings.AllowCorsFrom`](https://docs.mattermost.com/configure/integrations-configuration-settings.html#enable-cross-origin-requests-from) in the Mattermost server configuration. + +> [!NOTE] +> In particularly restrictive environments (e.g., VMs with strict network isolation), set `DOCKER_NETWORK=host` in the `calls-offloader` service environment so that job containers can reach the Mattermost server via its local address. + +### Custom Docker Registry + +If your air-gapped environment already has an internal Docker registry, you can point `install-offloader.sh` at it directly instead of using the local registry set up by the deployment script: + +```bash +sudo ./install-offloader.sh \ + --binary ./calls-offloader-linux-amd64 \ + --image-registry registry.internal.example.com/mattermost \ + --arch amd64 +``` + +The `--image-registry` flag sets the registry prefix used when the offloader pulls recorder and transcriber images for each job. + +## Other Calls Documentation + +- [Calls Overview](calls-deployment.md): Overview of deployment options and architecture +- [RTCD Setup and Configuration](calls-rtcd-setup.md): Comprehensive guide for setting up the dedicated RTCD service +- [Calls Metrics and Monitoring](calls-metrics-monitoring.md): Guide to monitoring Calls performance using metrics and observability +- [Calls Deployment on Kubernetes](calls-kubernetes.md): Detailed guide for deploying Calls in Kubernetes environments +- [Calls Troubleshooting](calls-troubleshooting.md): Detailed troubleshooting steps and debugging techniques +- [calls-offloader performance documentation](https://github.com/mattermost/calls-offloader/blob/master/docs/performance.md): Detailed performance tuning and monitoring recommendations \ No newline at end of file diff --git a/source/administration-guide/configure/calls-overview.rst b/source/administration-guide/configure/calls-overview.rst new file mode 100644 index 00000000000..0d41150ba80 --- /dev/null +++ b/source/administration-guide/configure/calls-overview.rst @@ -0,0 +1,27 @@ +Mattermost Calls Overview +========================= + +.. include:: ../../_static/badges/all-commercial.rst + :start-after: :nosearch: + +Mattermost Calls provides integrated audio calling and screen sharing capabilities within Mattermost channels. This section covers deployment, configuration, and management of Mattermost Calls. + +.. toctree:: + :maxdepth: 1 + :hidden: + + Deploy Mattermost Calls + RTCD Setup and Configuration + Calls Offloader Setup and Configuration + Calls Metrics and Monitoring + Deploy Calls on Kubernetes + Calls Troubleshooting + +**Calls Documentation:** + +* :doc:`Calls Deployment Overview ` - Main deployment overview and architecture guide for Mattermost Calls +* :doc:`RTCD Setup and Configuration ` - Real-time communication daemon setup for enterprise deployments +* :doc:`Calls Offloader Setup and Configuration ` - Configure call recording and transcription services +* :doc:`Calls Metrics and Monitoring ` - Performance monitoring with Prometheus and Grafana +* :doc:`Calls Deployment on Kubernetes ` - Kubernetes deployment guide for scalable Calls infrastructure +* :doc:`Calls Troubleshooting ` - Comprehensive troubleshooting guide for common issues diff --git a/source/administration-guide/configure/calls-rtcd-setup.md b/source/administration-guide/configure/calls-rtcd-setup.md new file mode 100644 index 00000000000..5057e2ec210 --- /dev/null +++ b/source/administration-guide/configure/calls-rtcd-setup.md @@ -0,0 +1,382 @@ +# RTCD Setup and Configuration + +```{include} ../../_static/badges/ent-plus.md +``` + +This guide provides detailed instructions for setting up, configuring, and validating a Mattermost Calls deployment using the dedicated RTCD service. + +## Prerequisites + +Before deploying RTCD, ensure you have: + +- A Mattermost Enterprise license +- A server or VM with sufficient CPU and network capacity (see the [Performance](calls-deployment.html#performance) section for sizing guidance) + +## Network Requirements + +The following network connectivity is required: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ServicePortsProtocolsSourceTargetPurpose
API (Calls plugin)80,443TCP (incoming)Mattermost clients (web/desktop/mobile)Mattermost instance (Calls plugin)To allow for HTTP and WebSocket connectivity from clients to Calls plugin. This API is exposed on the same connection as Mattermost, so there's likely no need to change anything.
RTC (Calls plugin or rtcd)8443UDP (incoming)Mattermost clients (Web/Desktop/Mobile) and calls-offloaderMattermost instance or rtcd serviceTo allow clients to establish connections that transport calls related media (e.g. audio, video). This should be open on any network component (e.g. NAT, firewalls) in between the instance running the plugin (or rtcd) and the clients joining calls so that UDP traffic is correctly routed both ways (from/to clients).
RTC (Calls plugin or rtcd)8443TCP (incoming)Mattermost clients (Web/Desktop/Mobile) and calls-offloaderMattermost instance or rtcd serviceTo allow clients to establish connections that transport calls related media (e.g. audio, video). This should be open on any network component (e.g. NAT, firewalls) in between the instance running the plugin (or rtcd) and the clients joining calls so that TCP traffic is correctly routed both ways (from/to clients). This can be used as a backup channel in case clients are unable to connect using UDP. It requires rtcd version >= v0.11 and Calls version >= v0.17.
API (rtcd)8045TCP (incoming)Mattermost instance(s) (Calls plugin)rtcd serviceTo allow for HTTP/WebSocket connectivity from Calls plugin to rtcd service. Can be exposed internally as the service only needs to be reachable by the instance(s) running the Mattermost server.
STUN (Calls plugin or rtcd)3478UDP (outgoing)Mattermost Instance(s) (Calls plugin) or rtcd serviceConfigured STUN servers(Optional) To allow for either Calls plugin or rtcd service to discover their instance public IP. Only needed if configuring STUN/TURN servers. This requirement does not apply when manually setting an IP or hostname through the ICE Host Override config option.
+ +## Installation and Deployment + +There are multiple ways to deploy RTCD, depending on your environment. We recommend the following order based on production readiness and operational control: + +### Bare Metal or VM Deployment (Recommended) + +This is the recommended deployment method for non-Kubernetes production environments, as it provides the best performance and operational control. For Kubernetes deployments, see the [Calls Deployment on Kubernetes](calls-kubernetes.md) guide. + +```{tip} +Looking for an automated setup? Check out these community-maintained [Calls Installation Scripts](https://github.com/bgardner8008/calls-install-scripts) for quick provisioning of the RTCD service on Ubuntu/Debian systems. +``` + +1. **Download and install the RTCD binary**: + + Download the latest release from the [RTCD GitHub repository](https://github.com/mattermost/rtcd/releases): + + ```bash + # Create the RTCD directory structure + sudo mkdir -p /opt/rtcd + + # Download the latest RTCD binary (adjust URL for your architecture) + # For Linux x86_64: + wget https://github.com/mattermost/rtcd/releases/latest/download/rtcd-linux-amd64 + + # Make the binary executable and move it to the installation directory + chmod +x rtcd-linux-amd64 + sudo mv rtcd-linux-amd64 /opt/rtcd/rtcd + ``` + + ```{note} + Replace `rtcd-linux-amd64` with the appropriate binary for your system architecture (e.g., `rtcd-linux-arm64` for ARM64 systems). The binary should be placed at `/opt/rtcd/rtcd` as this is the expected location referenced in systemd service files and other documentation. + ``` + +2. **Create a configuration file** (`/opt/rtcd/rtcd.toml`): + + Mattermost recommends using the official [config.sample.toml](https://github.com/mattermost/rtcd/blob/master/config/config.sample.toml) as a starting point. Download this file and use it as your base configuration. + +3. Create a dedicated user for the RTCD service: + + ```bash + sudo useradd --system --no-create-home --shell /bin/false mattermost + ``` + +4. Create the data directory and set ownership: + + ```bash + sudo mkdir -p /opt/rtcd/data/db + sudo chown -R mattermost:mattermost /opt/rtcd + ``` + +5. Create a systemd service file (`/etc/systemd/system/rtcd.service`): + + ```ini + [Unit] + Description=Mattermost RTCD Server + After=network.target + + [Service] + Type=simple + User=mattermost + Group=mattermost + ExecStart=/opt/rtcd/rtcd --config /opt/rtcd/rtcd.toml + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` + +6. Enable and start the service: + + ```bash + sudo systemctl daemon-reload + sudo systemctl enable rtcd + sudo systemctl start rtcd + ``` + +7. Check the service status: + + ```bash + sudo systemctl status rtcd + ``` + +### Docker Deployment + +Docker deployment is suitable for development, testing, or containerized production environments: + +1. Run the RTCD container with basic configuration: + + ```bash + docker run -d --name rtcd \ + -e "RTCD_LOGGER_ENABLEFILE=true" \ + -p 8443:8443/udp \ + -p 8443:8443/tcp \ + -p 8045:8045/tcp \ + mattermost/rtcd:latest + ``` + + ```{note} + If you optionally use the `RTCD_API_SECURITY_ALLOWSELFREGISTRATION` setting, please note that it defaults to `false`. If enabled, it allows anyone who can connect to the service on the API port (8045) to successfully initiate calls. Understand the security implications of this setting before enabling it. + ``` + +2. For debugging purposes, you can enable more detailed logging: + + ```bash + docker run -d --name rtcd \ + -e "RTCD_LOGGER_ENABLEFILE=true" \ + -e "RTCD_LOGGER_CONSOLELEVEL=DEBUG" \ + -p 8443:8443/udp \ + -p 8443:8443/tcp \ + -p 8045:8045/tcp \ + mattermost/rtcd:latest + ``` + + To view the logs: + + ```bash + docker logs -f rtcd + ``` + +You can also use a mounted configuration file instead of environment variables: + +```bash +docker run -d --name rtcd \ + -p 8045:8045 \ + -p 8443:8443/udp \ + -p 8443:8443/tcp \ + -v /path/to/config.toml:/rtcd/config/config.toml \ + mattermost/rtcd:latest +``` + +For a complete sample configuration file, see the [RTCD config.sample.toml](https://github.com/mattermost/rtcd/blob/master/config/config.sample.toml) in the official repository. + +### Kubernetes Deployment + +For detailed information on deploying RTCD in Kubernetes environments, including Helm chart configurations, resource requirements, and scaling considerations, see the [Calls Deployment on Kubernetes](calls-kubernetes.md) guide. + +## Configuration + +### RTCD Configuration File + +The RTCD service uses a TOML configuration file. Mattermost recommends using the official [config.sample.toml](https://github.com/mattermost/rtcd/blob/master/config/config.sample.toml) as your base configuration file. + +```{note} +A notable setting to be aware of is `ice_host_override` under the `[rtc]` section. You may need to configure this setting explicitly, particularly when RTCD is deployed behind NAT, in complex network topologies, or when automatic address discovery via STUN is unreliable. Setting `ice_host_override` directly to your server's public IP address or hostname is the preferred approach. +``` + +### TURN Configuration + +For clients behind strict firewalls, you may need to configure TURN servers. In the RTCD configuration file, reference your TURN servers as follows: + +```toml +[rtc] +# TURN server configuration + ice_servers = [ + { urls = ["turn:turn.example.com:3478"], username = "turnuser", credential = "turnpassword" } + ] + +``` + +We recommend using [coturn](https://github.com/coturn/coturn) for your TURN server implementation. + +### System Tuning + +For high-volume deployments, tune your Linux system: + +1. Add the following to `/etc/sysctl.conf`: + + ```bash + # Increase UDP buffer sizes + net.core.rmem_max = 16777216 + net.core.wmem_max = 16777216 + net.core.optmem_max = 16777216 + ``` + +2. Apply the settings: + + ```bash + sudo sysctl -p + ``` + +## Validation and Testing + +After deploying RTCD, validate the installation: + +1. **Check service status and version**: + + ```bash + curl http://YOUR_RTCD_SERVER:8045/version + # Should return a JSON object with service information + # Example: {"build_hash":"abc123","build_date":"2023-01-15T12:00:00Z","build_version":"0.11.0","goVersion":"go1.20.4"} + ``` + +2. **Test UDP connectivity**: + + Before testing, ensure the RTCD service is stopped, as it binds to the same port. + + ```bash + sudo systemctl stop rtcd + ``` + + On the RTCD server: + + ```bash + nc -l -u -p 8443 + ``` + + On a client machine: + + ```bash + nc -v -u YOUR_RTCD_SERVER 8443 + ``` + + Type a message and hit Enter on either side. If messages are received on both ends, UDP connectivity is working. + + Restart RTCD after the test: + + ```bash + sudo systemctl start rtcd + ``` + +3. **Test TCP connectivity** (if enabled): + + Similar to the UDP test, but remove the `-u` flag from both commands. + +4. **Monitor metrics**: + + Refer to [Calls Metrics and Monitoring](calls-metrics-monitoring.md) for setting up Calls metrics and monitoring. + +## Horizontal Scaling + +To scale RTCD horizontally: + +1. **Deploy multiple RTCD instances**: + + Deploy multiple RTCD servers, each with their own unique IP address. + +2. **Configure DNS record**: + + Set up a DNS record that points to multiple RTCD IP addresses: + + ```bash + rtcd.example.com. IN A 10.0.0.1 + rtcd.example.com. IN A 10.0.0.2 + rtcd.example.com. IN A 10.0.0.3 + ``` + +3. **Configure health checks**: + + Set up health checks to automatically remove unhealthy RTCD instances from DNS. + +4. **Configure Mattermost**: + + In the Mattermost System Console, set the **RTCD Service URL** to your DNS name (e.g., `rtcd.example.com`). + +When a call starts, the Mattermost server examines the available RTCD servers (via the configured DNS record) and starts the call on the RTCD server with the lowest CPU usage. All participants in the call will connect to that RTCD server; a single call cannot be shared across multiple servers. + +## Integration with Mattermost + +Once RTCD is properly set up and validated, configure Mattermost to use it: + +1. Go to **System Console > Plugins > Calls** + +2. Set the **RTCD Service URL** to your RTCD service address (either a single server or DNS load-balanced hostname). Ensure you provide any generated credentials formulated in the URI (e.g., `http://clientID:authKey@rtcd.local`). + +3. Save the configuration + +4. Test by creating a new call in any Mattermost channel + +5. Verify that the call is being routed through RTCD by checking the RTCD logs and metrics + +## Other Calls Documentation + +- [Calls Overview](calls-deployment.md): Overview of deployment options and architecture +- [Calls Offloader Setup and Configuration](calls-offloader-setup.md): Setup guide for call recording and transcription +- [Calls Metrics and Monitoring](calls-metrics-monitoring.md): Guide to monitoring Calls performance using metrics and observability +- [Calls Deployment on Kubernetes](calls-kubernetes.md): Detailed guide for deploying Calls in Kubernetes environments +- [Calls Troubleshooting](calls-troubleshooting.md): Detailed troubleshooting steps and debugging techniques + +For detailed Mattermost Calls configuration options, see the [Calls Plugin Configuration Settings](plugins-configuration-settings.rst#calls) documentation. \ No newline at end of file diff --git a/source/administration-guide/configure/calls-troubleshooting.md b/source/administration-guide/configure/calls-troubleshooting.md new file mode 100644 index 00000000000..b376587465d --- /dev/null +++ b/source/administration-guide/configure/calls-troubleshooting.md @@ -0,0 +1,32 @@ +# Troubleshooting Mattermost Calls + +```{include} ../../_static/badges/all-commercial.md +``` + +When troubleshooting Mattermost Calls, gathering the appropriate log files for analysis is critical. The required logs include: + +## RTCD logs + +The location of the RTCD log file is determined by the settings in your RTCD `config.toml` file. If you are running Calls using the integrated RTCD, these logs are included within the standard Mattermost server logs. + +## Client logs + +You can retrieve basic client logs by running the `/call logs` slash command to output the log for the most recent call. However, capturing the JavaScript console logs is often more helpful. The method for capturing these logs depends on your client: + +- **Desktop App:** While a call is in progress, navigate to **View > Developer Tools > Developer Tools for Call Widget**. Next to the **Filter** field, select **Verbose** from the log level drop-down menu to view the debug output. To save the log, right-click within the console and select **Save as...**. + +![Developer Tools for Call Widget](../../images/developer-tools-call-widget.png) + +- **Chrome browser:** Select the **More menu** (three vertical dots) next to the address bar, then go to **More Tools > Developer Tools**. Select the **Console** tab to view the output. Next to the **Filter** field, select **Verbose** from the log level drop-down menu to view the debug output. To save the log, right-click within the console and select **Save as...**. + +## Client statistics + +You can gather client call statistics by running the `/call stats` slash command. While you can run this command during an active call, it is best to run it immediately after a call concludes. This command provides valuable information regarding the negotiated connection between the client and RTCD, as well as statistics on media packets sent and lost. + +## Mattermost server logs + +The Mattermost server logs often contain entries related to the Calls plugin. If you are using an external RTCD instance, these logs are generally less critical than the RTCD and client logs. + +```{note} +Interpreting RTCD and client logs can be challenging, as the Interactive Connectivity Establishment (ICE) negotiation during call setup often logs expected and benign failure or error messages. We strongly recommend providing the complete set of logs to Mattermost Support for analysis. +``` \ No newline at end of file diff --git a/source/administration-guide/configure/plugins-configuration-settings.rst b/source/administration-guide/configure/plugins-configuration-settings.rst index fa3af1501ba..106aa58c617 100644 --- a/source/administration-guide/configure/plugins-configuration-settings.rst +++ b/source/administration-guide/configure/plugins-configuration-settings.rst @@ -510,7 +510,7 @@ ICE servers configurations - The configurations above, containing STUN and TURN servers, are sent to the clients and used to generate local candidates. - If hosting calls through the plugin (i.e. not using the |rtcd_service|) any configured STUN server may also be used to find the instance's public IP when none is provided through the |ice_host_override_link| option. -.. |rtcd_service| replace:: :ref:`rtcd service ` +.. |rtcd_service| replace:: :doc:`RTCD service ` **Example** @@ -758,7 +758,7 @@ Call recording quality .. note:: - This setting is applicable only to self-hosted deployments. - - The quality setting will affect the performance of the job service and the file size of recordings. Refer to the :ref:`deployment section ` for more information. + - The quality setting will affect the performance of the job service and the file size of recordings. Refer to the :ref:`deployment section ` for more information. .. config:setting:: enable-pluginscalltranscriptions :displayname: Enable call transcriptions (Plugins - Calls) @@ -810,7 +810,7 @@ Transcriber model size .. note:: - This setting is applicable only to self-hosted deployments. - - This setting is available starting in plugin version 0.22. The model size setting will affect the performance of the job service. Refer to the :ref:`configure call recordings, transcriptions, and live captions ` documentation for more information. + - This setting is available starting in plugin version 0.22. The model size setting will affect the performance of the job service. Refer to the :ref:`configure call recordings, transcriptions, and live captions ` documentation for more information. .. config:setting:: call-transcriber-threads :displayname: Call transcriber threads (Plugins - Calls) @@ -834,7 +834,7 @@ Call transcriber threads .. note:: - This setting is applicable only to self-hosted deployments. - - The call transcriber threads setting will affect the performance of the job service. Refer to the :ref:`configure call recordings, transcriptions, and live captions ` documentation for more information. This setting is available starting in plugin version 0.26.2. + - The call transcriber threads setting will affect the performance of the job service. Refer to the :ref:`configure call recordings, transcriptions, and live captions ` documentation for more information. This setting is available starting in plugin version 0.26.2. .. config:setting:: enable-pluginslivecaptions :displayname: (Experimental) Enable live captions (Plugins - Calls) diff --git a/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-ddil-operations.rst b/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-ddil-operations.rst index 46240abe206..5c9b4268071 100644 --- a/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-ddil-operations.rst +++ b/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-ddil-operations.rst @@ -69,7 +69,7 @@ This hybrid deployment architecture provides optimal collaboration in both conne - **Object Storage:** File uploads, images, and attachments are stored outside the application node on an :doc:`S3-compatible store ` or network/local storage, hosted locally within your tactical network. - - **Recording Instance:** ``calls-offloader`` :ref:`job service ` to offload heavy processing tasks from Mattermost Calls to self-hosted infrastructure within your tactical network, such as recordings, transcriptions, and live captioning. *(Optional)* + - **Recording Instance:** ``calls-offloader`` :ref:`job service ` to offload heavy processing tasks from Mattermost Calls to self-hosted infrastructure within your tactical network, such as recordings, transcriptions, and live captioning. *(Optional)* - **Self-hosted integrations:** :doc:`Custom apps, plugins, and webhooks ` can be deployed within your tactical network. *(Optional - not shown)* @@ -103,8 +103,8 @@ Self-hosted audio & screensharing Effective collaboration at the tactical edge requires all voice and screen sharing capabilities remain operational without reliance on the internet or third-party services. Deploy :doc:`Mattermost Calls ` in a self-hosted configuration, including: -- The :ref:`rtcd service ` for scalable, low-latency media routing hosted on-premises. Run multiple ``rtcd`` nodes for redundancy. -- The :ref:`calls offloader ` service offloads heavy processing tasks like recording, transcription and live captioning to a locally hosted compliance-approved job server. +- The :ref:`rtcd service ` for scalable, low-latency media routing hosted on-premises. Run multiple ``rtcd`` nodes for redundancy. +- The :ref:`calls offloader ` service offloads heavy processing tasks like recording, transcription and live captioning to a locally hosted compliance-approved job server. High availability and fault tolerance ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-mission-partner.rst b/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-mission-partner.rst index 62f43107472..f177eef8d57 100644 --- a/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-mission-partner.rst +++ b/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-mission-partner.rst @@ -82,7 +82,7 @@ The deployment architecture includes the following components: - **Object Storage:** File uploads, images, and attachments are stored outside the application node on an :doc:`S3-compatible store ` or an NFS (Network File System) server. `Azure Blob Storage `_ can be used, but needs an S3-compatible proxy for Mattermost to interface with. - - **Recording Instance:** ``calls-offloader`` :ref:`job service ` to offload heavy processing tasks from Mattermost Calls, such as recordings, transcriptions, and live captioning, to local infrastructure or private cloud. *(Optional)* + - **Recording Instance:** ``calls-offloader`` :ref:`job service ` to offload heavy processing tasks from Mattermost Calls, such as recordings, transcriptions, and live captioning, to local infrastructure or private cloud. *(Optional)* - **Integration framework:** :doc:`Custom apps, plugins, and webhooks ` can be deployed for real-time data integrations and alerting. *(Optional - not shown)* @@ -161,8 +161,8 @@ Sovereign audio & screensharing Deploy :doc:`Mattermost Calls ` in a self-hosted configuration to ensure voice and screen sharing capabilities remain operational without reliance on the internet, and that media traffic does not traverse non-compliant third-party services. -- The :ref:`rtcd service ` for scalable, low-latency media routing hosted on-premises. Run multiple ``rtcd`` nodes for redundancy. -- The :ref:`calls offloader ` service offloads heavy processing tasks like recording, transcription and live captioning to a compliance-approved job server. +- The :ref:`rtcd service ` for scalable, low-latency media routing hosted on-premises. Run multiple ``rtcd`` nodes for redundancy. +- The :ref:`calls offloader ` service offloads heavy processing tasks like recording, transcription and live captioning to a compliance-approved job server. Compliance and retention ~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-sovereign-collaboration.rst b/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-sovereign-collaboration.rst index e196cb05d79..bd94aa5efaf 100644 --- a/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-sovereign-collaboration.rst +++ b/source/deployment-guide/reference-architecture/deployment-scenarios/deploy-sovereign-collaboration.rst @@ -57,7 +57,7 @@ The deployment architecture includes the following components: - **Object Storage:** File uploads, images, and attachments are stored outside the application node on an :doc:`S3-compatible store ` or an NFS (Network File System) server. `Azure Blob Storage `_ can be used, but needs an S3-compatible proxy for Mattermost to interface with. - - **Recording Instance:** ``calls-offloader`` :ref:`job service ` to offload heavy processing tasks from Mattermost Calls, such as recordings, transcriptions, and live captioning, to enterprise-controlled infrastructure or private cloud. *(Optional)* + - **Recording Instance:** ``calls-offloader`` :ref:`job service ` to offload heavy processing tasks from Mattermost Calls, such as recordings, transcriptions, and live captioning, to enterprise-controlled infrastructure or private cloud. *(Optional)* - **Self-hosted integrations:** :doc:`Custom apps, plugins, and webhooks ` can be deployed within the enterprise boundary. *(Optional - not shown)* @@ -88,8 +88,8 @@ Sovereign audio & screensharing Data sovereignty compliance may require that all voice and screen sharing traffic remain within enterprise-controlled infrastructure and does not traverse third-party services. Deploy :doc:`Mattermost Calls ` in a self-hosted configuration to ensure that Microsoft Teams users and Mattermost users collaborate without media ever leaving the sovereign network. -- The :ref:`rtcd service ` for scalable, low-latency media routing hosted on-premises. Run multiple ``rtcd`` nodes for redundancy. -- The :ref:`calls offloader ` service offloads heavy processing tasks like recording, transcription and live captioning to a compliance-approved job server. +- The :ref:`rtcd service ` for scalable, low-latency media routing hosted on-premises. Run multiple ``rtcd`` nodes for redundancy. +- The :ref:`calls offloader ` service offloads heavy processing tasks like recording, transcription and live captioning to a compliance-approved job server. Compliance and retention ~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/deployment-guide/server/containers/install-docker.rst b/source/deployment-guide/server/containers/install-docker.rst index 5a34687849e..bf301bcce8f 100644 --- a/source/deployment-guide/server/containers/install-docker.rst +++ b/source/deployment-guide/server/containers/install-docker.rst @@ -206,7 +206,7 @@ Looking for a way to evaluate Mattermost on a single local machine using Docker? - This local image is self-contained (i.e., it has an internal database and works out of the box). Dropping a container using this image removes data and configuration as expected. You can see the :doc:`configuration settings ` documentation to learn more about customizing your trial deployment. - **Preview Mode** shouldn't be used in a production environment, as it uses a known password string, contains other non-production configuration settings, has email disabled, keeps no persistent data (all data lives inside the container), and doesn't support upgrades. - - If you are planning to use the calling functionality in **Preview Mode** on a non-local environment, you should ensure that the server is running on a secure (HTTPs) connection and that the :ref:`network requirements ` to run calls are met. + - If you are planning to use the calling functionality in **Preview Mode** on a non-local environment, you should ensure that the server is running on a secure (HTTPs) connection and that the :ref:`network requirements ` to run calls are met. 1. Install `Docker `__. diff --git a/source/deployment-guide/server/preparations.rst b/source/deployment-guide/server/preparations.rst index c7bf899431c..f7524911601 100644 --- a/source/deployment-guide/server/preparations.rst +++ b/source/deployment-guide/server/preparations.rst @@ -10,7 +10,7 @@ This guide outlines the key preparation steps required before installing the Mat Review software and hardware requirements Set up an NGINX proxy - Configure Mattermost Calls + Configure Mattermost Calls Set up TLS Use an image proxy @@ -18,7 +18,7 @@ Before installing Mattermost Server, review the following preparation requiremen * :doc:`Review software and hardware requirements ` - Ensure your system meets the minimum requirements for Mattermost deployment. * :doc:`Set up an NGINX proxy ` - Configure NGINX as a reverse proxy for enhanced security and performance. -* :doc:`Configure Mattermost Calls ` - Set up real-time communication capabilities for voice and video calls. +* :doc:`Configure Mattermost Calls ` - Set up real-time communication capabilities for voice and video calls. * :doc:`Set up TLS ` - Enable secure communication with SSL/TLS encryption. * :doc:`Use an image proxy ` - Configure image proxy for enhanced privacy and security. diff --git a/source/end-user-guide/collaborate/make-calls.rst b/source/end-user-guide/collaborate/make-calls.rst index 974877e6b7e..6d483399642 100644 --- a/source/end-user-guide/collaborate/make-calls.rst +++ b/source/end-user-guide/collaborate/make-calls.rst @@ -10,9 +10,12 @@ Using a web browser, the desktop app, or the mobile app, you can `join a call <# - All Mattermost customers can start, join, and participate in 1:1 audio calls with optional screen sharing. - For group calls up to 50 concurrent users, Mattermost Enterprise, Professional, or Mattermost Cloud is required. - - Enterprise customers can also `record calls <#record-a-call>`__, enable :ref:`live text captions ` during calls, and `transcribe recorded calls <#transcribe-recorded-calls>`__. We recommend that Enterprise self-hosted customers looking for group calls beyond 50 concurrent users consider using the :ref:`dedicated rtcd service `. + - Enterprise customers can also `record calls <#record-a-call>`__, enable :ref:`live text captions ` during calls, and `transcribe recorded calls <#transcribe-recorded-calls>`__. We recommend that Enterprise self-hosted customers looking for group calls beyond 50 concurrent users consider using the :doc:`dedicated RTCD service `. - Mattermost Cloud users can start calling right out of the box. For Mattermost self-hosted deployments, System admins need to enable and configure the plugin :ref:`using the System Console `. +.. include:: ../../_static/badges/academy-calls.rst + :start-after: :nosearch: + Join a call ----------- diff --git a/source/images/calls-channel-enable-disable.png b/source/images/calls-channel-enable-disable.png new file mode 100644 index 00000000000..a2751779dad Binary files /dev/null and b/source/images/calls-channel-enable-disable.png differ diff --git a/source/images/developer-tools-call-widget.png b/source/images/developer-tools-call-widget.png new file mode 100644 index 00000000000..54cc679cc5d Binary files /dev/null and b/source/images/developer-tools-call-widget.png differ