Skip to content

DOC-1276 Add Iceberg metrics#1090

Merged
JakeSCahill merged 3 commits intomainfrom
iceberg-metrics
Apr 29, 2025
Merged

DOC-1276 Add Iceberg metrics#1090
JakeSCahill merged 3 commits intomainfrom
iceberg-metrics

Conversation

@JakeSCahill
Copy link
Contributor

@JakeSCahill JakeSCahill commented Apr 28, 2025

Description

Resolves https://redpandadata.atlassian.net/browse/DOC-1276
Review deadline: April 29

This pull request adds a new section to the modules/reference/pages/public-metrics-reference.adoc file, documenting metrics related to the Iceberg REST client and Iceberg translation. These metrics provide detailed insights into HTTP requests, data processing, and error handling for the Iceberg integration.

Additions to Iceberg REST client metrics:

  • Added metrics to track active HTTP requests (GET and PUT), total requests, and bytes sent/received, categorized by role.
  • Included counters for specific endpoint interactions (e.g., commit_table_update, create_namespace, load_table) and their failure rates.
  • Documented metrics for transport errors, request timeouts, and OAuth token requests.

Additions to Iceberg translation metrics:

  • Introduced metrics for data processing, such as bytes decompressed, raw bytes processed, and rows added to Parquet files, categorized by redpanda_namespace and redpanda_topic.
  • Added counters for invalid records, DLQ file creation, and completed translator executions.

Page previews

Checks

  • New feature
  • Content gap
  • Support Follow-up
  • Small fix (typos, links, copyedits, etc)

Summary by CodeRabbit

  • Documentation
    • Added a new section detailing Iceberg metrics to the public metrics reference, including gauges and counters for monitoring Iceberg REST client activity and translation processes. Metrics cover HTTP requests, errors, data throughput, and translation outcomes, with clear labeling for easier tracking and analysis. No previous content was modified or removed.

@JakeSCahill JakeSCahill requested a review from a team as a code owner April 28, 2025 14:48
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Apr 28, 2025

Walkthrough

A new "Iceberg metrics" section has been introduced to the public metrics reference documentation. This addition documents a comprehensive set of new gauges and counters related to the Iceberg REST client and translation processes, including metrics for HTTP request tracking, error counts, data throughput, and translation outcomes. The metrics are categorized and labeled for clarity, and the update extends the documentation without altering or removing any existing content.

Changes

File(s) Change Summary
modules/reference/pages/public-metrics-reference.adoc Added a new "Iceberg metrics" section with detailed documentation for new REST client and translation metrics.

Poem

In the land of metrics, fresh snowflakes fall,
Iceberg stats now tracked, we’re counting them all!
From RESTful requests to translation bytes,
Gauges and counters light up the nights.
Rabbits rejoice as the numbers grow—
Documented clearly, so everyone knows!
🐇📊


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 23e0e3c and 2e3f99d.

📒 Files selected for processing (1)
  • modules/reference/pages/public-metrics-reference.adoc (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • modules/reference/pages/public-metrics-reference.adoc

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@netlify
Copy link

netlify bot commented Apr 28, 2025

Deploy Preview for redpanda-docs-preview ready!

Name Link
🔨 Latest commit 2e3f99d
🔍 Latest deploy log https://app.netlify.com/sites/redpanda-docs-preview/deploys/680fc45740866f00088ccc0b
😎 Deploy Preview https://deploy-preview-1090--redpanda-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
modules/reference/pages/public-metrics-reference.adoc (2)

232-520: Group and document gauge vs counter metrics more clearly
All Iceberg REST client gauges are listed first, followed by counters, but there’s no visual divider beyond the repeated ---. Consider adding subsection headings (e.g., “=== Gauges” and “=== Counters”) or a brief usage note for each group to improve scannability.


522-624: Normalize label ordering and description phrasing in translation metrics
Most translation metrics list labels as redpanda_namespace, then redpanda_topic, but redpanda_iceberg_translation_invalid_records places redpanda_cause first. Please choose a consistent label order—e.g., primary entity first (namespace, topic) followed by cause.
Also, descriptions vary in phrasing (“Number of bytes consumed post-decompression…” vs “Number of raw, potentially compressed bytes…”). Standardize on a concise template such as “Total number of ” and consider shortening or removing parenthetical examples to keep entries uniform.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2e1e840 and f4a7964.

📒 Files selected for processing (1)
  • modules/reference/pages/public-metrics-reference.adoc (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (3)
  • GitHub Check: Redirect rules - redpanda-docs-preview
  • GitHub Check: Header rules - redpanda-docs-preview
  • GitHub Check: Pages changed - redpanda-docs-preview


=== redpanda_iceberg_rest_client_num_request_timeouts

Total number of catalog requests that could no longer be retried because they timed out. This may occur if the catalog is not responding.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused about when we say "number" v when we say "total number". What is the rule for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It just comes from the code, and we haven't made an effort to make them consistent yet. There are several instances, so I'd rather handle those in a separate pull request.

Copy link
Contributor

@asimms41 asimms41 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just a couple of questions around consistency.

Co-authored-by: Angela Simms <102690377+asimms41@users.noreply.github.com>
@JakeSCahill JakeSCahill requested a review from asimms41 April 28, 2025 18:09
Copy link
Contributor

@asimms41 asimms41 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JakeSCahill JakeSCahill merged commit 85b5d77 into main Apr 29, 2025
8 checks passed
@JakeSCahill JakeSCahill deleted the iceberg-metrics branch April 29, 2025 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants