[WIP] perf: optimize boot time with lazy imports and combined DB queries#378
Open
mrosseel wants to merge 156 commits intobrickbots:mainfrom
Open
[WIP] perf: optimize boot time with lazy imports and combined DB queries#378mrosseel wants to merge 156 commits intobrickbots:mainfrom
mrosseel wants to merge 156 commits intobrickbots:mainfrom
Conversation
- build.yml: single build + Cachix push + unstable channel updates - release.yml: manual release workflow for stable/beta channels Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The SD image module provides filesystems, but toplevel builds need a minimal stub to evaluate successfully. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Required for NixOS module system to accept devMode setting. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Required when module has both options and config sections. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replaces FIXME placeholders with actual SRI hashes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Uses Pi5 runner when RUNNER_LABELS variable is set, falls back to ubuntu with QEMU emulation otherwise. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Filter to only Pi 4B device tree (CM4 incompatible with our overlays) - Use shorthand DTS syntax for PWM overlay Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Untracked file was excluded from Nix flake source tree, causing "No module named 'PiFinder.sys_utils_base'" on SD card boot. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add camera overlay (imx477) to netboot config.txt via flake.nix - Fix sys_utils import in main.py to use utils.get_sys_utils() - Add hip_main.dat fetch to pifinder-src.nix for starfield plotting - Add dma_heap udev rule for libcamera/picamera2 access - Fix shared memory naming in solver.py (remove leading /) - Add DNS nameservers for netboot environment - Document power control scripts in CLAUDE.md Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add runtimeCameraSelection option to hardware.nix (default: true) - SD image includes config.txt with "include camera.txt" directive - Users can edit camera.txt and reboot to switch cameras - Supported cameras: imx296, imx290 (imx462), imx477 - Fix cameraDriver scope in hardware.nix (moved to top-level let) - Add sudoers rules for systemctl stop/start pifinder.service - Add DMA heap udev rule for libcamera video group access - Netboot config sets cameraType = "imx477" for HQ camera dev Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Refactor sys_utils modules to use common base class - Add sys_utils_nixos.py for NixOS-specific implementations - Add get_sys_utils() detection in utils.py for platform selection - Add flake.lock for reproducible builds - Add NetworkManager config to networking.nix - Add deploy-image-to-nfs.sh for netboot development workflow Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update build.yml CI workflow - Fix fonts.py import - Fix marking_menus.py formatting - Add missing import to preview.py - Simplify objects_db.py - Add catalog_imports improvements - Update pifinder_objects.db Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Switch to NFSv4 with caching disabled (noac, actimeo=0) - Disable auto-optimise-store in devMode (hard links fail on NFS) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ServerAliveInterval/CountMax to prevent timeout during transfers - Use rsync -R (relative) to preserve directory structure correctly Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Comets.txt is downloaded at runtime and must be in a writable location, not the read-only Nix store. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Extend eth0 wait to 30 seconds with debug output - Wait for link carrier before DHCP - Add DHCP retries (3 attempts) - Add LIBCAMERA_IPA_MODULE_PATH to pifinder service environment Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Restore SUBSYSTEM=="pwm" udev rule that was accidentally removed. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Turns on keypad LEDs during sysinit for early visual boot feedback. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- boot-splash.c: displays welcome image with scanning animation - Starts at sysinit, stops when pifinder.service starts - Much faster than Python splash Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove nixos-hardware module (saves 659MB linux-firmware) - Fetch nixos-rebuild at runtime (saves ~500MB llvm/nix deps) - Remove git from systemPackages (nix has built-in git for flakes) Target: ~150MB vs current 1.7GB
- Remove default packages (vim, nano, etc) - Disable polkit, udisks2, speechd - Should reduce closure significantly
NetworkManager-vpnc alone has 1.1GB closure (webkitgtk, llvm, etc). Disable all NM plugins for bootstrap - we just need WiFi.
# Conflicts: # nixos/services.nix # pifinder-build.json # python/PiFinder/ui/software.py
Derives version string from flake's self.shortRev so local nix builds show "nix-<sha>" instead of stale CI-committed values. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add includeCatalogImages param to mkPifinderSystem - Add images.migration output (SD image without catalog images) - Delete images.bootstrap tarball derivation - Rename bootstrap-tarball.yml → migration-image.yml - Update release.yml to build migration image instead of tarball Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… RAM) - Revert includeCatalogImages/images.migration from flake.nix - Workflow builds tarball from SD image (excluding catalog_images) - Add migration tarball step to release.yml Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unmount and detach loop devices from previous runs before rm, preventing "Device or resource busy" failures on self-hosted runner. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Self-hosted runner retains /tmp files across runs. Move all cleanup (mounts, loop devices, tarballs) to a dedicated step at the start. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After creating the release, the workflow now checks out the migration branch and updates the hardcoded SHA256 and size in software.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migration tarball now uses a slimmed-down NixOS profile (migration.nix) that only includes SSH, avahi, and filesystem tools. The full PiFinder system (Python, samba, cedar-detect, libcamera, GPSD) is downloaded from cachix on first boot via pifinder-first-boot.service. CI builds the full system first and pushes to cachix, then builds the migration SD image and injects the full system store path into /var/lib/pifinder/first-boot-target. Expected tarball size reduction: ~1.4GB -> ~400-600MB. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitHub only discovers workflows from the default branch, so the standalone migration-tarball.yml never triggered. Merge it as a build-migration-tarball job in build.yml that reuses the full system store path from build-native/build-emulated. Only runs on nixos branch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Disable nix channels/registry/nixPath to avoid pulling nixpkgs source (~189 MB) into the closure - Strip NetworkManager VPN plugins (openconnect, stoken, gtk3 deps) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- nix.registry/nixPath with mkForce to prevent flake self-reference from pulling nixpkgs source (~186 MB) - system.disableInstallerTools removes nixos-rebuild-ng which pulls Python 3.13 (~110 MB) - environment.defaultPackages = mkForce [] removes perl (~59 MB) - Swap vim (42 MB) for nano (3 MB) Expected savings: ~390 MB uncompressed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace SD image build/mount/extract cycle with direct closure assembly. This eliminates ext4 filesystem overhead, reducing tarball size to fit in Pi 4's 924 MB tmpfs. Also factors out configTxt and adds migration-boot-firmware package to deduplicate boot partition assembly across profiles. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Nix store paths are read-only and root-owned; rm -rf without sudo fails on the copied closure paths. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge upstream changes including: - IMU integrator selection (classic/quaternion) - Eq-mount support under IMU dead-reckoning - Chinese locale and font support - Harris Globular Cluster catalog - T9 input support - PAM-based web auth (replacing su) - Stellarium+ Mobile support - Various bug fixes NixOS adjustments: - Add quaternion (numpy-quaternion) to python-packages.nix - Add quaternion to mypy ignore list in pyproject.toml - Add get_initialized_solved_dict() lost during merge resolution - Fix formatting in integrator.py - Keep requirements.txt and version.txt deleted (managed by nix) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reduces kernel-relative Event Loop time from ~48s to ~24s on RPi4.
Test plan
🤖 Generated with Claude Code