RoSE-SLAM — KV260 ORB-SLAM2 Acceleration

FPGA-accelerated ORB front-end and stereo depth for ORB-SLAM2 on the Xilinx Kria KV260, derived from the open-source acSLAM project and adapted to the KV260 platform. The design implements a resize + FAST pipeline in PL (ORB front-end) and a Semi-Global Matching (SGM) stereo accelerator, driven from Linux on the KV260 via AXI DMA. Software integrates with ORB-SLAM2 to offload feature extraction and stereo processing.

FPT’25 Design Competition

This repository is the submission package for the FPT’25 Design Competition — SLAM Algorithm FPGA Acceleration and Optimization Track.
Our result: 3rd Place in the SLAM track.

Overview

• ORB front-end: accelerates ORB feature extraction (resize + FAST) on KV260 PL.
  • Data path: PS → AXI DMA (MM2S) → resize → FAST_extractor → AXI DMA (S2MM) → PS.
  • Two AXI DMAs: one config-only stream (MM2S), one data stream (MM2S + S2MM).
• Stereo SGM: dense disparity accelerator for KITTI / USB camera stereo pipelines.
• Software paths: Python via PYNQ (notebook) or C++ via CYNQ; full ORB_SLAM2 integration provided.

Folder Layout

• code/HW — HLS / Vitis HLS sources and scripts
  • code/HW/hls/resize — resize + config fan-out HLS IP.
  • code/HW/hls/FAST_extractor — FAST detector HLS IP.
  • code/HW/hls/sgm — SGM accelerator for KITTI stereo (semiglobalbm_accel).
  • code/HW/hls/sgm_usbcam — SGM accelerator variant for USB camera stereo (semiglobalbm_accel_usbcam).
• code/SW — Software
  • code/SW/ORB_SLAM2_FPGA — ORB‑SLAM2 integration (CMake, examples, config).
  • code/SW/cynq — CYNQ C++ framework (docs) used by the integration.
  • code/SW/cynq_test — Low-level CYNQ demos and docs (e.g., orb_cynq_demo.cpp, orb_cynq_demo.md).
  • code/SW/pynq_test — Minimal PYNQ notebook test (ORB.ipynb, sample image).
• code/Vivado — Vivado block design scripts, projects and bitstreams
  • code/Vivado/kv260_mono.tcl — mono ORB front-end design.
  • code/Vivado/kv260_stereo_kitti.tcl — stereo KITTI design with SGM.
  • code/Vivado/kv260_stereo_usbcam.tcl — stereo USB camera design with SGM.
  • code/Vivado/Bitstreams/ — built bitstreams and .hwh files (e.g. mono_kitti.bit, stereo_kitti-108-3.bit, stereo_usbcam.bit).
• verification_evo — Trajectory evaluation toolkit (vendored evo)
  • Local copy of the evo toolkit (APE/RPE) with helpers under verification_evo/contrib.
  • Example (APE, TUM format):
    • python3 verification_evo/evo/main_ape.py tum code/SW/ORB_SLAM2_FPGA/KeyFrameMono/KeyFrameTrajectory04.txt /path/to/kitti_gt_folder/kitti_04_gt.txt -as -p
  • Batch helpers: see verification_evo/contrib/batching_evo.sh and utilities for KITTI conversions.

Results (KV260)

• Resource Utilization

	Utilization	Available	Utilization %
CLB LUTs	74364	117120	63.49%
FF	54597	234240	23.31%
BRAM	138.5	114	96.18%
DSP	245	1248	19.63%

• End-to-end SLAM performance (on sequence 04, without GUI)
  • KITTI Monocular: 10.8 FPS
  • KITTI Stereo: 2.10 FPS

Prerequisites

• Vivado Design Suite 2024.2.2
• Board: Xilinx Kria KV260 (xck26-sfvc784-2LV-c) running Ubuntu 22.04
• Libraries for ORB_SLAM2: Pangolin, OpenCV (Tested on 4.5.4), Eigen3.
• CYNQ installed (Tested on commit 39f0acc) and discoverable via pkg-config (pkg-config --cflags --libs cynq).
• Optional: PYNQ.
• Clone with submodules: git clone --recursive https://github.com/BJUT-CS316-LAB/RoSE-SLAM.git (needed for code/SW/cynq and code/HW/hls/sgm/Vitis_Libraries; if already cloned run git submodule update --init --recursive).

Download KITTI

• Download data_odometry_gray.zip from: https://www.cvlibs.net/datasets/kitti/eval_odometry.php
• Unzip:
  • unzip data_odometry_gray.zip -d ~/KITTI
• Set your data root to the sequences directory:
  • ~/KITTI/data_odometry_gray/sequences/ (contains 00/, 01/, ...)
  • If using run_kitti.sh: set DATA_ROOT to that path (or pass --data-root).

Rebuild Steps

• Step 1 — Rebuild HLS IP in code/HW/hls

  • Open Vivado HLS shell and run:
    • cd code/HW/hls/resize && vitis_hls -f script.tcl
    • cd ../FAST_extractor && vitis_hls -f script.tcl
    • cd ../sgm && vitis_hls -f script.tcl
    • cd ../sgm_usbcam && vitis_hls -f script.tcl
  • The scripts target KV260 and export IP to the Vivado IP catalog (see code/HW/README.md for details and Vitis Vision / OpenCV requirements for SGM).

• Step 2 — Rebuild Vivado projects in code/Vivado

  • Open a Vivado 2024.2 Tcl shell and run:
    • cd code/Vivado
    • Mono ORB front-end:
      • vivado -mode batch -source kv260_mono.tcl
    • Stereo KITTI (default design used by ORB_SLAM2_FPGA):
      • vivado -mode batch -source kv260_stereo_kitti.tcl
    • Stereo USB camera:
      • vivado -mode batch -source kv260_stereo_usbcam.tcl
  • Each script creates a project (mono_proj, stereo_kitti_proj, or stereo_usbcam_proj) with block design design_1. Generate the bitstream in Vivado (GUI or Tcl, e.g. launch_runs impl_1 -to_step write_bitstream on the generated wrapper).
  • Copy the resulting bitstreams (.bit/.hwh, e.g. design_1_wrapper.bit) into code/Vivado/Bitstreams/ and/or update software config to point to the desired path.

Run Options

• Option A — Quick PYNQ notebook test

  • Copy your desired bitstream (for example code/Vivado/Bitstreams/mono_kitti.bit(.hwh) or stereo_kitti-108-3.bit(.hwh)) next to the notebook or update the path inside it.
  • Open and run code/SW/pynq_test/ORB.ipynb on the KV260 to program PL and test with 000000.png.

• Option B — Low-level CYNQ demo (single image test)

  • Build: cd code/SW/cynq_test/ && g++ -O3 -std=c++17 orb_cynq_demo.cpp -o orb_cynq_demo.run $(pkg-config --cflags --libs opencv4 cynq)
  • Run (example, mono ORB front-end): sudo ./orb_cynq_demo.run ../../Vivado/Bitstreams/mono_kitti.bit ./000000.png
  • The demo programs the bitstream via FPGA Manager and reports detected keypoints and timings.

• Option C — ORB_SLAM2 with FPGA front-end

  • Install dependencies (OpenCV, Pangolin, Eigen3).
  • Build:
    • cd code/SW/ORB_SLAM2_FPGA
    • ./build.sh
  • Example (KITTI, Monocular):
    • ./Examples/Monocular/mono_kitti.run Vocabulary/ORBvoc.bin Examples/Monocular/KITTI04-12.yaml /path/to/KITTI/sequences/04/
  • Example (KITTI, Stereo):
    • ./Examples/Stereo/stereo_kitti.run Vocabulary/ORBvoc.bin Examples/Stereo/KITTI04-12.yaml /path/to/KITTI/sequences/04/
  • Helper: code/SW/ORB_SLAM2_FPGA/run_kitti.sh batches multiple sequences and renames outputs.

Config Notes

• Default bitstream path used by the ORB‑SLAM2 integration is configured in code/SW/ORB_SLAM2_FPGA/fpga_config.yaml (key FPGA.BitstreamPath). The repository default points to the stereo KITTI design:
  • ../../Vivado/Bitstreams/stereo_kitti-108-3.bit
• You can override the config file location by setting the ORB_FPGA_CONFIG environment variable when launching the ORB_SLAM2 binaries, or by editing fpga_config.yaml directly.
• If the KV260 has no display, disable the viewer when building/running ORB_SLAM2.

Verify the results with APE and RPE

• Install evo in your verification environment:

  • pip install evo

• Monocular (TUM-formatted trajectories):

  • Use tum mode when your trajectory is already in TUM format (time tx ty tz qx qy qz qw).
  • Single-run APE (example, save xz plot):
    • evo_ape tum ./KeyFrame/KeyFrameTrajectory04.txt ./kitti_gt_folder/kitti_04_gt.txt -as --plot_mode xz --save_plot results_mono/track_map_mono/xz/traj_mono_xz_04.png
  • Single-run RPE (example, save xz plot):
    • evo_rpe tum ./KeyFrame/KeyFrameTrajectory04.txt ./kitti_gt_folder/kitti_04_gt.txt -as --plot_mode xz --save_plot results_mono/track_map_mono/xz/traj_mono_rpe_xz_04.png
  • To generate additional projections, change --plot_mode to xy or yz (the repository batch helper creates xz/, xy/, yz/ folders).
  • Batch helper (process sequences 00..10):
    • cd verification_evo/contrib && ./batching_evo_KITTI_mono.sh

• Stereo (KITTI-formatted keyframe poses / 3x4 ground truth):

  • Use kitti mode when keyframe/ground-truth files are KITTI 3x4 poses (no TUM timestamps).
  • Single-run APE (example, save xz plot):
    • evo_ape kitti ./KeyFrameStereo/CameraTrajectory04.txt ./real/04.txt -as --plot_mode xz --save_plot results_stereo/track_map_stereo/xz/traj_stereo_xz_04.png
  • Single-run RPE (example, save xz plot):
    • evo_rpe kitti ./KeyFrameStereo/CameraTrajectory04.txt ./real/04.txt -as --plot_mode xz --save_plot results_stereo/track_map_stereo/xz/traj_stereo_rpe_xz_04.png
  • To generate additional projections, change --plot_mode to xy or yz (batch helper saves all three projections to xz/, xy/, yz/).
  • Batch helper (process sequences 00..10):
    • cd verification_evo/contrib && ./batching_evo_KITTI_stereo.sh

• Notes:

  • The batch helpers save APE/RPE numeric summaries to results_mono/APE_RPE_results_kitti_mode_mono.txt and results_stereo/APE_RPE_results_kitti_mode_stereo.txt respectively.
  • If your monocular output is in KITTI format (3x4), convert it to TUM before using tum mode; the mono batch script will call the converter when needed.
  • --plot_mode choices: xz, xy, yz. Use --save_plot <file> to write the plot image to disk.