Skip to content

refactor: Replace JBang with javax.tools.JavaCompiler for UDF compilation#1969

Open
velo wants to merge 5 commits intomainfrom
refactor/replace-jbang-with-java-compiler
Open

refactor: Replace JBang with javax.tools.JavaCompiler for UDF compilation#1969
velo wants to merge 5 commits intomainfrom
refactor/replace-jbang-with-java-compiler

Conversation

@velo
Copy link
Copy Markdown
Collaborator

@velo velo commented Mar 30, 2026

Summary

  • Replace JBang external process with in-process javax.tools.JavaCompiler for compiling Flink UDF .java scripts
  • Add Mima (eu.maveniverse.mima) for resolving //JDEPS Maven dependencies (replaces JBang's //DEPS)
  • Auto-detect UDF files by class hierarchy (no shebang marker needed)
  • Old JBang shebang lines are automatically stripped for backward compatibility

Changes

  • New: UdfCompiler — compiles UDF sources and packages into JAR using javax.tools.JavaCompiler
  • New: MavenDependencyResolver — parses //JDEPS and resolves artifacts via Mima embedded Maven resolver
  • New: JavaUdfPreprocessor — replaces JBangPreprocessor, auto-detects .java files extending Flink UDF classes
  • Deleted: JBangRunner, JBangPreprocessor and their tests
  • Updated: Dockerfiles (removed JBang image stage and PATH), templates (removed shebang), Spring wiring

Test plan

  • 30 new unit tests (UdfCompilerTest, JavaUdfPreprocessorTest, MavenDependencyResolverTest)
  • All 94 sqrl-cli tests pass
  • Full build passes (207 integration tests, 0 failures)
  • Container IT (JavaUdfContainerIT) — requires Docker image rebuild

🤖 Generated with Claude Code

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 30, 2026

Codecov Report

❌ Patch coverage is 61.93182% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 13.27%. Comparing base (76b9af1) to head (799da83).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...i/src/main/java/com/datasqrl/util/UdfCompiler.java 67.76% 33 Missing and 6 partials ⚠️
...ava/com/datasqrl/util/MavenDependencyResolver.java 36.58% 25 Missing and 1 partial ⚠️
.../src/main/java/com/datasqrl/util/SqrlInjector.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1969      +/-   ##
============================================
+ Coverage     13.11%   13.27%   +0.16%     
  Complexity      780      780              
============================================
  Files           592      593       +1     
  Lines         16848    16908      +60     
  Branches       1971     1974       +3     
============================================
+ Hits           2210     2245      +35     
- Misses        14435    14459      +24     
- Partials        203      204       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@velo
Copy link
Copy Markdown
Collaborator Author

velo commented Mar 30, 2026

UDF Container Test Performance

JBang (before) JavaCompiler (after) Speedup
JavaUdfContainerIT 111.5s 19.0s 5.8x faster

From CircleCI logs — same test, same Docker container setup, just the UDF compilation path changed.

@velo velo requested a review from ferenc-csaky March 30, 2026 15:03
@velo velo marked this pull request as ready for review April 2, 2026 23:26
@velo velo force-pushed the refactor/replace-jbang-with-java-compiler branch from 9ad8c4d to ed79a39 Compare April 7, 2026 00:03
velo added 5 commits April 7, 2026 14:36
…tion

Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
…R at runtime

Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
…ove TOCTOU

Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
@velo velo force-pushed the refactor/replace-jbang-with-java-compiler branch from e41c3f2 to 799da83 Compare April 7, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant