Skip to content

[Issue #1305] standardize file naming#1308

Open
gengdy1545 wants to merge 3 commits intopixelsdb:masterfrom
gengdy1545:feature/standardizeFileName
Open

[Issue #1305] standardize file naming#1308
gengdy1545 wants to merge 3 commits intopixelsdb:masterfrom
gengdy1545:feature/standardizeFileName

Conversation

@gengdy1545
Copy link
Collaborator

@gengdy1545 gengdy1545 commented Mar 16, 2026

Summary

Standardize all .pxl file naming conventions to resolve Issue #1305
(file naming issues in FileWriterManager). This is a subtask of Issue #1288
([pixels-retina, pixels-index, cpp] create new file) and lays the naming
foundation required by the Storage GC file-grouping and rewrite mechanism.

Background

Each write path (Retina CDC, batch load, compaction) used its own ad-hoc naming
scheme, making it impossible for Storage GC to reliably extract virtualNodeId
(for grouping files to rewrite), file type (for GC eligibility), and hostName
(for routing to the correct Retina service).

Changes

  • Add PixelsFileNameUtils: Establishes a unified .pxl file name format
    <hostName>_<timestamp>_<count>_<virtualNodeId>_<type>.pxl as the single
    source of truth. Provides builder methods for all write paths
    (buildRetinaFileName, buildOrderedFileName, buildCompactFileName,
    buildSingleFileName, buildCopyFileName) and parser methods
    (extractHostName, extractVirtualNodeId, extractFileType, isGcEligible).
  • Add NetUtils: Consolidates scattered hostname resolution logic from
    multiple modules into a shared utility with a well-defined fallback chain.
  • Refactor CompactExecutor: Groups input files by virtualNodeId before
    compaction to preserve the delete_ts monotonicity invariant required by
    Storage GC; skips copy-type and unrecognized-format files; fixes in-place
    mutation of numRowGroupInBlock/compactLayout.
  • Update all write paths (FileWriterManager, IndexedPixelsConsumer,
    SimplePixelsConsumer) to use the unified naming API.
  • Simplify RetinaUtils: Replace inline hostname extraction with
    PixelsFileNameUtils.extractHostName() and isGcEligible().

Breaking Change

Existing files with the old naming format are incompatible with the new parser.
Migration or fresh deployment is required before upgrading.

Closes #1305
Part of #1288

@gengdy1545 gengdy1545 requested review from AntiO2 and bianhq March 16, 2026 15:15
@gengdy1545 gengdy1545 self-assigned this Mar 16, 2026
@gengdy1545 gengdy1545 added the enhancement New feature or request label Mar 16, 2026
@gengdy1545 gengdy1545 added this to the Real-time CRUD milestone Mar 16, 2026
@gengdy1545 gengdy1545 linked an issue Mar 16, 2026 that may be closed by this pull request
@gengdy1545 gengdy1545 requested a review from AntiO2 March 17, 2026 03:19
Copy link
Collaborator

@AntiO2 AntiO2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[pixels-retina] file naming issues in FileWriterManager

2 participants