Add TLS/mTLS End-to-End Integration Tests for OTLP/OTAP Receivers and Exporters

[lalitb]

Summary

We have good component-level TLS tests (handshakes, certificate verification, hot-reload), but no end-to-end coverage that covers the full network path. The TLS path on OTLP/OTAP receivers is untested with real traffic, and exporter-side TLS is only validated against mock servers without a running pipeline.

Target flow:

  External Client → [TLS/mTLS] → OTLP/OTAP Receiver → Pipeline → OTLP/OTAP Exporter → [TLS/mTLS] → Backend                                         

Goal

Add integration tests (feature-gated: experimental-tls) that:

1. Generate self-signed CA/server/client certs on the fly.
2. Start a pipeline with TLS-enabled OTLP/OTAP receiver(s).
3. Send real OTLP/OTAP data from a TLS/mTLS client over the network.
4. Verify the data traverses the pipeline to an OTLP/OTAP exporter and is observed by a TLS/mTLS backend (mock tonic server or capture exporter).

Scope / Decisions

• Port discovery: pre-bind ephemeral ports before building the config, or expose a test-only hook to read TcpListener::local_addr() after bind.
• Assertions: use either a capture exporter (test sink) or a mock tonic OTLP/OTAP backend to assert receipt.
• Test location: keep under crates/otap/tests unless a separate tests/ crate is justified.
• Signals: exercising one signal (logs/metrics/traces) is sufficient if it uses real OTLP/OTAP payloads.
• Protocol combinations (OTLP receiver): cover gRPC-only, HTTP-only, and dual-protocol.
• Hot reload: include a certificate rotation test and verify connections/traffic still succeed post-rotation.
• Cover both TLS-only and mTLS.
• CI: must pass in rust-ci.yml on Linux with --all-features.
• Validation scope: verify data reaches the backend, not payload content correctness.

Notes

• Leverage existing TLS utilities (tls_utils.rs, mtls_tests.rs, tls_stream.rs) but add full pipeline wiring. Can also be reused once TLS support for syslog lands (TLS support for Syslog #1260).
• The experimental-tls flag should be removed once the integration tests are in place (Remove experimental-tls feature flag and make TLS support built-in by default #1742)

---

Early proposal, haven't brought this up in community meetings yet. Thoughts welcome.