Skip to content

Explicitly build dynamic op libraries for wheel packaging#51

Open
czgdp1807 wants to merge 1 commit intogoogle:masterfrom
czgdp1807:so-build
Open

Explicitly build dynamic op libraries for wheel packaging#51
czgdp1807 wants to merge 1 commit intogoogle:masterfrom
czgdp1807:so-build

Conversation

@czgdp1807
Copy link
Contributor

Problem

When building the struct2tensor wheel, patchelf warnings occur during the stamp_wheel step,

patchelf: getting info about 'struct2tensor/ops/_decode_proto_map_op.so': No such file or directory

The stamp_wheel function expects all six dynamic op library files (.so) to be present in struct2tensor/ops/ before attempting to process them, but they were not being built and copied from Bazel's output directory.

Root Cause

With the upgrade to Protobuf 4.25.6, the build system changed:

  • Before (Protobuf 3.21.9): The protobuf.bzl provided public macros (cc_proto_library, py_proto_library, proto_gen rule with gen_cc attribute) that automatically compiled .proto files to C++ .pb.cc/.pb.h files. These were used by dynamic library targets.
  • After (Protobuf 4.25.6): These public macros were removed and replaced with internal underscored versions (_proto_gen, internal_py_proto_library). The old proto_gen rule also removed the gen_cc attribute, making C++ generation implicit and harder to trigger.

Bottom line: Protobuf 4.25.6 removed the public build rule APIs that struct2tensor's proto dependencies relied on. Without access to cc_proto_library from protobuf.bzl, the proto files in struct2tensor couldn't be compiled to C++, preventing the dynamic libraries from being built.

Solution

Add an explicit build_dynamic_libraries() function in build_common.sh that:

  1. Builds each of the six dynamic op library targets using bazel build
  2. Copies the built .so files from bazel-bin/struct2tensor/ops/ to struct2tensor/ops/
  3. Runs before the bazel run :build_pip_package step to ensure files are staged for wheel creation

This ensures the .so files are present and valid before stamp_wheel attempts to patch them.

Files Changed

  • build_common.sh: Added build_dynamic_libraries() function and integrated it into the build flow
  • struct2tensor/tools/docker_build/build_manylinux.sh: Removed trailing whitespace

Testing

Wheel builds successfully with all dynamic libraries properly staged and patched.

Add explicit build and copy steps for dynamic op libraries (.so files) in
build_common.sh. This ensures all six dynamic library targets are built and
copied to struct2tensor/ops/ before the wheel is packaged.

Previously, the stamp_wheel step in build_manylinux.sh would fail with
patchelf errors ("No such file or directory") because the .so files were
expected in struct2tensor/ops/ but were not being copied from bazel-bin.

This fix:
- Explicitly builds each dynamic library target
- Copies the built .so files from bazel-bin to the staging directory
- Runs before bazel run :build_pip_package to ensure files are present

Fixes the missing _decode_proto_map_op.so, _decode_proto_sparse_op.so and
related dynamic library warnings during wheel packaging.
@czgdp1807
Copy link
Contributor Author

Here are the build logs,

.
.
.
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_decode_proto_map_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_decode_proto_map_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_decode_proto_map_op.so
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_decode_proto_sparse_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_decode_proto_sparse_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_decode_proto_sparse_op.so
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_run_length_before_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_run_length_before_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_run_length_before_op.so
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_equi_join_any_indices_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_equi_join_any_indices_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_equi_join_any_indices_op.so
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_equi_join_indices_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_equi_join_indices_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_equi_join_indices_op.so
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_parquet_dataset_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_parquet_dataset_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_parquet_dataset_op.so
+ rm /struct2tensor/dist/struct2tensor-0.49.0.dev0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+ wheel version
wheel 0.46.3
+ wheel pack /tmp/tmp.hWCK9OOEW6 --dest-dir /struct2tensor/dist
Repacking wheel as /struct2tensor/dist/struct2tensor-0.49.0.dev0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl...OK

@czgdp1807
Copy link
Contributor Author

After merging this PR and #50 in my master branch locally, I tried both bazel build /... (inside docker container) and docker build --no-cache --build-arg TF_SERVING_VERSION_GIT_BRANCH=2.17.0 -t local/s2t_tf_serving .. Both are working fine. So @vkarampudi you can go ahead, merge both the PRs into master and start the release process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant