VIO and SLAM

Notes on visual-inertial odometry (VIO), SLAM (simultaneous localization and mapping), and the estimation layer that lets a drone fly indoors without GPS. Assumes comfort with linear algebra, rotations, and sensors, but no prior state-estimation background.

Core take: VIO answers “where am I relative to where I started”; SLAM answers “where am I in a map I am building, and do I recognize this place.” VIO is odometry — it integrates motion and drifts without bound. SLAM adds a persistent map and loop closure, which is what bounds the drift. Everything else in the field is detail on top of this split.

The second organizing fact: shipped autonomy stacks do not run one monolithic SLAM system. They run sparse VIO for the flight controller (smooth, fast, drifting) and a separate dense mapper for the planner (consistent, slow, allowed to jump when corrected). Most integration pain — and most of the architecture work left to us in the indoor capture project — lives at the seam between the two.

The estimation problem

A multirotor’s controller needs, at a few hundred Hz: orientation, angular rate, velocity, and position, plus their uncertainties. Outdoors, GNSS (satellite positioning — GPS and its siblings) anchors position and the rest follows from cheap sensor fusion. Indoors nothing anchors anything: the IMU (inertial measurement unit — 3-axis accelerometer plus gyroscope) alone gives orientation and, briefly, motion — but position from an IMU means double-integrating noisy acceleration, and the error grows quadratically. A consumer-grade MEMS (microfabricated silicon) IMU is off by meters within seconds.

So GPS-denied flight is a state-estimation problem: fuse the IMU with a sensor that observes the world outside the vehicle (camera, lidar, ToF — time-of-flight) and use the static geometry it sees as the position reference the IMU lacks — estimating everything relative to where the vehicle started.

Two properties frame everything downstream:

  • Some states are observable, some are not. Gravity makes roll and pitch absolutely observable — they do not drift. Yaw and position have no absolute reference indoors; they are only ever estimated relative to the starting pose, and their error accumulates with distance traveled. Good VIO drifts 0.3–1% of trajectory length: after 100 m of indoor flight, expect to be 0.3–1 m off. No amount of odometry polish removes this — only recognizing a previously seen place (loop closure) or an external reference does.
  • The output frame is local. The origin is wherever the estimator initialized, gravity-aligned. Nothing ties it to a building floor plan, a previous session’s map, or true north; aligning to any of those is a separate problem (registration) with its own machinery.

Frame conventions — NED/FRD (north-east-down world, forward-right-down body) on the flight-control side, ENU/FLU (east-north-up, forward-left-up) on the ROS side — are a standing source of integration bugs: an orientation must be converted at both its world and body ends, and a bridge that fixes only one end flies with mirrored yaw. MAVROS and the PX4 ROS 2 interface library do this correctly; hand-rolled bridges must get it right themselves.

VIO: why a camera plus an IMU

Visual-inertial odometry fuses a camera (mono, stereo, or more) with an IMU to estimate 6-DoF pose at high rate. The pairing works because the two sensors fail in complementary ways:

IMU Camera
Rate 200 Hz–1 kHz 15–60 Hz
Short-term excellent blur, rolling shutter
Long-term drifts in seconds drift-free relative geometry
Absolute scale observable (measures m/s²) unobservable with one camera
Failure modes bias wander, vibration low texture, low light, fast rotation
“Loses lock” never routinely

The IMU bridges between camera frames and makes metric scale observable for monocular setups — accelerations tell you how large the world is. The camera anchors the IMU’s bias states and velocity. Neither sensor alone survives drone speeds and rotation rates; together they are the cheapest, lightest full-pose sensor that exists, which is why every phone AR (augmented reality) stack (ARKit, ARCore) and every small-drone autonomy stack is built on VIO.

Two non-obvious consequences of the fusion math:

  • VIO needs excitation. Scale and bias become observable only under accelerated motion. Constant velocity or a long pure hover degrades the estimate; most systems require a deliberate motion (a small jerk or figure) during initialization before the output is trustworthy.
  • The IMU noise model is load-bearing. The estimator co-estimates slowly-wandering accelerometer and gyro biases, weighted by a stochastic model (white noise density + bias random walk, classically identified from an Allan variance log). Feeding a simulator’s broken IMU model into real VIO code produces confidently wrong output — the exact failure class our B1.4 IMU audit measured in Pegasus (accel bias silently dropped, gyro tau copy-pasted).

Anatomy of a VIO pipeline

Every system decomposes the same way:

Frontend — feature tracking. Detect sparse interest points (FAST corners — Features from Accelerated Segment Test; ORB features — Oriented FAST and Rotated BRIEF) and track them across frames, either by descriptor matching or KLT (Kanade–Lucas–Tomasi) optical flow. Reject outliers with RANSAC (random sample consensus — fit on random minimal subsets, keep the largest inlier set) on epipolar geometry. The frontend is where the environment decides how much information the pipeline gets at all — texture, lighting, and motion blur act here. Every stage after it only processes what the frontend delivered; none can recover what it lost.

IMU preintegration. Between two camera frames there are hundreds of IMU samples. Preintegration (Forster et al.) compresses them into a single relative-motion constraint whose value can be cheaply re-linearized when the bias estimate changes, so the backend never touches raw samples.

Backend — the estimator. Two schools:

  • Filtering (EKF — extended Kalman filter — and specifically the MSCKF). The multi-state constraint Kalman filter (Mourikis & Roumeliotis 2007) keeps a sliding window of past camera poses in an EKF state and uses each feature track as a constraint across them — landmarks never enter the state, so cost stays fixed and low. This is the lineage of phone AR and of Qualcomm’s closed mvVISLAM that ships as qVIO on VOXL (see VOXL 2 Mini): ~30 Hz pose at a fraction of one core.
  • Sliding-window optimization. Nonlinear least squares over the last N keyframes plus landmarks, older states marginalized out. More accurate, more CPU. VINS-Fusion, ORB-SLAM3’s VI mode, Basalt, and NVIDIA’s cuVSLAM are here; OpenVINS is a well-documented hybrid on the MSCKF side.

Output contract. Pose + velocity + covariance at IMU or camera rate, in a gravity-aligned local frame, plus a quality/health signal. The covariance and the health signal matter as much as the pose: the consumer (flight controller, mapper) must know when to stop trusting the estimate.

The filter-vs-optimization tradeoff is mostly compute-vs-accuracy, and it has converged in practice: filters on embedded targets and phones, optimization where there is CPU/GPU headroom, with the accuracy gap narrowing as MSCKF variants improved.

Compute. VIO is cheap — it ran on 2014 phone SoCs (Project Tango), and qVIO takes ~10–15% of one QRB5165 core because the frontend rides the DSP/ISP (digital / image signal processors — the SoC’s camera-adjacent co-processors). Budget roughly one core-equivalent on a modern ARM board for feature-based VIO at VGA (640×480) / 30 Hz; cost scales with feature count × frame rate (resolution barely matters — accuracy comes from geometry and sub-pixel tracking, so VGA is standard). VIO is never the reason to buy a bigger companion computer; mapping and planning are.

What the hardware must provide

VIO quality is decided before any algorithm runs. The requirements, in order of how often violating them ruins a system:

  • Global shutter. Rolling shutter skews geometry under motion; it can be modeled, but every serious tracking camera avoids it instead. This is why VOXL tracking cameras are VGA global-shutter fisheyes rather than anything higher-resolution: for odometry, geometry beats pixels.
  • Wide field of view. A fisheye keeps features in view through aggressive rotation and guarantees parallax somewhere in the image. Resolution is nearly irrelevant — features are tracked to sub-pixel accuracy anyway.
  • Hardware timestamping and sync. The estimator needs camera exposure midpoints and IMU samples on one clock, to well under a millisecond. USB cameras with software timestamps are how hobby VIO projects die.
  • Rigid mounting and calibration. Camera intrinsics, camera–IMU extrinsics (rotation to ~0.1°, translation to ~mm), and the camera–IMU time offset. Kalibr is the standard offline tool; some estimators (VINS, OpenVINS) refine extrinsics and time offset online. Miscalibration is the classic silent accuracy killer — everything runs, the numbers are just worse than they should be.
  • Vibration isolation. Multirotor frames resonate at frequencies that alias into the IMU band; soft-mounting the IMU/camera assembly is standard.

Integrated platforms (VOXL, Skydio, phone AR) win not on algorithms but on having done this sensor engineering once, correctly, at the factory — per-unit calibrated, synced, and vibration-characterized.

SLAM: odometry plus a map plus recognition

Simultaneous localization and mapping takes odometry and adds three things:

  1. A map as a first-class product — a sparse landmark cloud, a graph of keyframes, or a dense volume, depending on what the map is for.
  2. Place recognition. Recognize that the current view matches a previously visited place — classically bag-of-visual-words (DBoW2), increasingly learned global descriptors (NetVLAD lineage) — then verify geometrically and compute the relative pose between now and then.
  3. Global correction. The new loop-closure constraint contradicts the drifted odometry chain, so re-optimize. The standard machinery is a pose graph: nodes are keyframe poses, edges are relative-pose constraints from odometry and loop closures; solving it (g2o, Ceres, GTSAM) snaps the whole trajectory and map back into consistency.

The modern unifying view is the factor graph: variables (poses, velocities, biases, landmarks) connected by factors (preintegrated IMU, feature observations, loop closures, GPS when present). SLAM is incremental inference over that graph — iSAM2 (incremental smoothing and mapping) is the incremental solver — and a filter is just one way of approximating it. Dellaert’s Factor Graphs for Robot Perception is the canonical treatment.

Under the same umbrella:

  • Relocalization — recover from tracking loss, or wake up inside an existing map, via the same place-recognition machinery.
  • Map reuse — localize today against yesterday’s map. Once the map is frozen this stops being SLAM and becomes plain localization, which is cheaper and more robust; multi-session systems (ORB-SLAM3’s atlas, ARKit/ARCore persistent anchors) blur the line.

What loop closure buys, concretely: drift stops scaling with distance traveled and starts scaling with distance since the last revisit. For a coverage mission that repeatedly crosses its own path — exactly the indoor capture pattern — that is the difference between centimeter- and meter-level map consistency.

Compute. Unlike VIO, the SLAM additions have unbounded and spiky costs: descriptor extraction on every keyframe is a steady tax, place-recognition queries are milliseconds, but pose-graph solves grow with trajectory length and spike to hundreds of milliseconds on a long mission. All of it belongs on a background core — never in the control path, which only ever consumes the smooth odometry stream.

The taxonomy

Axes along which systems differ, and where the field settled:

  • Sparse vs dense. Sparse systems track a few hundred feature points — enough to estimate the sensor’s own motion, but their “map” is a scatter of isolated landmarks, useless for collision checking. Dense/volumetric systems model surfaces and free space — maps a planner can consume. Deployed stacks run one of each, split as described below.
  • Indirect vs direct. Indirect matches features; direct methods (LSD-SLAM, DSO — Direct Sparse Odometry) minimize photometric error on raw pixels, which handles low-texture scenes better but hates rolling shutter, auto-exposure, and lighting change. Feature-based won in deployed systems; direct survives in research and as hybrid refinement.
  • Visual vs lidar. Lidar-inertial odometry (FAST-LIO2, LIO-SAM) measures geometry directly: robust to lighting and texture, indifferent to darkness, heavier and costlier. On small indoor drones the weight budget usually forces visual; on anything that can lift a spinning or solid-state lidar, lidar-inertial is the more robust default and is what the survey-grade indoor platforms (Flyability, Emesent, Exyn) fly. RGB-D (color plus per-pixel depth) and ToF sit between: direct depth like lidar, camera-class range and noise (see the ToF-vs-lidar explainer in VOXL 2 Mini).
  • Classical vs learned. Learned components are replacing modules — features (SuperPoint), matching (SuperGlue/LightGlue), place recognition, monocular depth priors — while the estimation core of every deployed system remains classical geometry plus optimization. End-to-end learned SLAM (DROID-SLAM lineage, and Gaussian-splatting SLAM such as SplaTAM/MonoGS) produces impressive offline results and is not yet what anyone flies on an embedded board.
  • Semantic/metric-semantic. Attaching object/room labels to the map (Kimera, Hydra scene graphs). Directly relevant to capture products — “this is a door, this is a room” is the structure a house viewer needs — but a layer above the estimation problem, not part of it.

Inputs: images, depth maps, point clouds

A common misconception is that SLAM consumes depth maps. Only one family does; what each family takes in:

Family Input Depth is…
Monocular / VI SLAM RGB images (+ IMU) an output, triangulated from parallax
Stereo SLAM two images computable, often skipped — features matched across views directly
RGB-D SLAM color + per-pixel depth map an input, measured by the sensor
Lidar SLAM point clouds the input, in a different sampling pattern
Dense mapping (any stack) posed depth, always required — free space can only be ray-cast through measured range

RGB-D in more depth, since it is the natural indoor modality. The depth map comes from one of three sensor technologies:

  • Time-of-flight — modulated IR (infrared) light, per-pixel phase → range. Compact, fast, low resolution; the VOXL ToF module (PMD/IRS2975C, see VOXL 2 Mini) is this, ~5 m indoor range.
  • Structured light — project an IR pattern, decode deformation (original Kinect, iPhone FaceID). Precise close-in, poor beyond a few meters, defeated by sunlight.
  • Active stereo — plain stereo matching helped by a projected IR texture (RealSense D4xx). Degrades to passive stereo where the projector doesn’t reach; the most flexible of the three.

What direct depth buys the estimator: metric scale with no motion required (the motion-based initialization monocular VIO depends on becomes unnecessary), depth for every pixel including textureless walls (the indoor killer for visual methods), and dense maps essentially free — KinectFusion-lineage systems fuse depth frames straight into a TSDF. RTAB-Map and ORB-SLAM3’s RGB-D mode are the standard open systems; ElasticFusion the classic dense research line.

The costs, and why drones still fly on VIO-plus-depth rather than RGB-D SLAM: active IR range tops out around 4–6 m indoors and collapses outdoors in sunlight; dark, absorbing, and reflective surfaces return no or false depth; rolling-shutter color cameras and unsynced depth/color pairs break fast motion; and the sensors cost weight and power. So the typical drone architecture keeps the fisheye+IMU VIO as the pose backbone and treats depth (ToF, stereo, lidar) as a mapping sensor consuming VIO poses — RGB-D SLAM proper shines on slower platforms: handheld scanners, ground robots, and close-range inspection.

Dense mapping: the other half of the stack

The planner does not want poses; it wants to know what space is occupied, free, and — critically for exploration — unknown. Dense mapping consumes posed depth (from stereo, ToF, lidar, or back-projected depth images, as in our B2.1 lesson) and maintains a volumetric world model:

  • Occupancy grids / octrees (OctoMap, 2D or 3D): per-voxel occupied/free/unknown via ray-casting from the sensor origin. The free/unknown distinction is what makes frontier-based exploration possible at all — a frontier is the free/unknown boundary.
  • TSDF (truncated signed distance field): per-voxel signed distance to the nearest surface, fused over many depth frames. Averages sensor noise into clean surfaces; meshes well (marching cubes). The reconstruction-friendly representation.
  • ESDF (Euclidean signed distance field): per-voxel distance to the nearest obstacle, derived from the TSDF or occupancy. This is the planner-friendly representation — collision margin queries and trajectory optimization gradients come for free.

voxblox (ETH-ASL — ETH Zurich’s Autonomous Systems Lab; TSDF→ESDF on CPU) is the classic implementation and is what the VOXL SDK vendors inside voxl-mapper; nvblox is the GPU-resident successor in Isaac ROS; RTAB-Map packages SLAM-plus-dense-mapping as one ROS-native system. This layer, not odometry, is what sizes the computer: voxblox at useful resolution occupies 1–2 CPU cores on its own, and nvblox exists precisely because volumetric fusion is the part worth a GPU.

The architectural point everything else rests on: the dense map is only as consistent as the poses feeding it. Odometry drift smears walls into double surfaces — the corruption mode the B1.3/B2.1 injected-pose negative controls demonstrate deliberately. Systems handle this either by keeping missions short relative to drift, or by rebuilding/deforming the volume after loop closures (submapping: fuse locally rigid chunks, re-pose the chunks on correction).

Where the compute lives: CPU vs GPU/NPU/DSP

Classical SLAM is, first-order, a CPU workload — but the accurate statement is per stage: the pixel-facing stages offload well and increasingly do, while the estimation core resists offload and stays on the CPU.

Stays on CPU:

  • The solvers. MSCKF updates, sliding-window solves, pose-graph and bundle adjustment via Ceres/g2o/GTSAM. These are sparse, irregular linear algebra: small variable blocks, data-dependent sparsity, iterative relinearization with branching. GPUs want large, regular, dense batches; a 20-keyframe window solve is none of that. GPU bundle adjustment exists as research (MegBA and kin); no deployed stack uses it.
  • Graph bookkeeping — keyframe management, place-recognition indices, covisibility graphs: pointer-chasing code.

Routinely offloaded:

  • The frontend — the one genuinely image-parallel estimation stage. Feature detection, pyramid building, KLT tracking: qVIO runs this on the Hexagon DSP/ISP (the whole trick behind its ~10% core cost), cuVSLAM on CUDA, phones on their ISP blocks.
  • Depth computation. Stereo matching is embarrassingly parallel and often never touches the host: the RealSense D435i computes depth on its own onboard ASIC (application-specific chip) and ships finished depth maps over USB; VOXL’s DFS (depth-from-stereo) runs on DSP/GPU; Isaac ROS runs SGM (semi-global matching) on the GPU.
  • Volumetric fusion. TSDF integration is per-voxel parallel — nvblox is GPU-resident end to end.
  • Learned components. SuperPoint-class features, learned place recognition, monocular depth: NPU (neural processing unit) / GPU by construction. “Features on the NPU, solver on the CPU” is becoming the embedded pattern.

Sizing consequence: a companion computer needs a couple of strong CPU cores regardless of how large its GPU/NPU is — and conversely, a GPU buys mapping and learned perception, not better odometry. Fully-GPU systems exist only in the learned lineage (DROID-SLAM runs everything, including its update operator, on GPU), and those are not embedded-deployable today.

Systems worth knowing by name

System Kind License Note
OpenVINS VIO, MSCKF GPL-3.0 Cleanest documented MSCKF; the reference open VIO and the best codebase to learn from
VINS-Fusion VIO + loop closure GPL-3.0 Sliding-window optimization; battle-tested on drones; optional GPS fusion
ORB-SLAM3 Full VI-SLAM GPL-3.0 Accuracy benchmark king; multi-map atlas, relocalization; heavier, touchier to integrate
Basalt VIO + mapping BSD-3 Fast optimization-based VIO from TUM; permissive license
Kimera VI-SLAM + semantic mesh BSD-2 MIT-SPARK; metric-semantic; research-grade
cuVSLAM / Isaac ROS Visual SLAM Stereo VI-SLAM, GPU proprietary, free The Jetson-native option; loop closure included; see Isaac ecosystem
qVIO / mvVISLAM VIO, MSCKF closed Qualcomm; ships on VOXL; ~30 Hz, ~10% of one core; odometry only — no loop closure
RTAB-Map RGB-D/stereo SLAM + dense map BSD-3 “Just give me a map in ROS”; long-term memory for large spaces
FAST-LIO2 Lidar-inertial odometry GPL-2.0 The LIO default; tightly-coupled, very robust indoors
LIO-SAM Lidar-inertial SLAM BSD-3 Factor-graph LIO with loop closure
ARKit / ARCore VIO + relocalization closed Phone AR; the largest-scale deployed VIO; useful as a capture-pose source (see 3D photo reconstruction)
DROID-SLAM Learned dense SLAM BSD-3 End-to-end learned lineage; GPU-hungry, offline/near-line

Practical selection logic: on VOXL, qVIO is already integrated and calibrated — use it and manage drift (ModalAI’s SDK also carries an OpenVINS-based server as the open successor path). On Jetson, cuVSLAM. On a PC with ROS 2 and an RGB-D or stereo camera, RTAB-Map for maps or OpenVINS/VINS-Fusion for odometry. If the platform lifts a lidar, FAST-LIO2 and stop worrying about texture and light. GPL licenses matter the moment estimation code links into a shipped product — one reason vendors keep VIO closed or BSD.

Integration with the flight stack

The autopilot’s estimator (PX4 EKF2, ArduPilot’s EKF3) is not SLAM and not even full VIO — it is a fusion filter over IMU, baro, mag, GNSS, and, GPS-denied, an external odometry input. The division of labor: VIO estimates pose; the autopilot EKF makes it flight-control-grade — outlier gating, consistent covariance, smooth high-rate output, graceful degradation.

External estimates enter via MAVLink ODOMETRY (pose + twist + covariances + explicit source frames) or, on the PX4 ROS 2 path, the uXRCE-DDS (PX4’s direct DDS bridge, see ROS, from zero) VehicleVisualOdometry topic — the interface the VOXL voxl-hitl-vio-server exposes on UDP 14560 for simulator injection. Frame fields (MAV_FRAME_LOCAL_FRD, MAV_FRAME_BODY_FRD) must be set correctly; the EKF will happily fuse mirrored velocities.

The part of the contract that is easy to miss is reset and jump semantics. When SLAM loop-closes or relocalizes, its pose estimate jumps — sometimes by meters. A position controller fed that jump executes a step response into a wall. Hence the two-frame architecture, standardized in ROS as REP-105:

  • odom frame — continuous, smooth, drifting. Odometry lives here; the controller consumes this and nothing else.
  • map frame — globally consistent, allowed to jump. Loop-closed SLAM poses live here; the planner and the dense map consume this. The correction is published as a slowly-updating map→odom transform rather than as a rewritten robot pose.

ODOMETRY carries a reset_counter so consumers can detect discontinuities; PX4’s EKF gates innovation spikes and can reset its own origin. In practice the safe pattern is: feed the autopilot only smooth odometry, keep loop-closed consistency for the map and planner, and treat any frame jump as a planning event, never a control event.

Failure modes

The reflexes to build, roughly ordered by how often each one bites indoors:

  • Texture-poor surfaces. Blank walls and ceilings are the indoor killer — no corners, no tracks, no constraint. Wide FOV mitigates (something textured is usually visible somewhere); lidar/ToF is immune.
  • Low light and motion blur. Exposure time trades noise against blur; indoor light levels force that tradeoff into the bad region. Fast yaw at long exposure destroys feature tracks — our B1.2 lesson’s 360° spin is a gentle version of a genuinely hostile maneuver.
  • Pure rotation. No translation means no parallax means depth unobservable. Brief rotations coast on the IMU; sustained rotation-in-place degrades the map.
  • Dynamic scenes. Everything above assumes a static world. People, pets, and doors violate it. Frontends reject moving features as outliers, which works while the moving things are a minority of the image — a scene that is mostly in motion defeats the voting and drags the estimate with it.
  • Reflective and transparent surfaces. Mirrors and glass present geometrically false features (and false depth to active sensors) — a genuinely unsolved practical problem for indoor capture.
  • Initialization. Monocular VIO needs motion before scale converges; taking off on an unconverged estimator is a classic crash. Stereo and direct depth largely remove this.
  • Calibration and sync errors. Silent, systematic, and worst of all plausible — the system works, drifts twice as fast as it should, and nothing flags it. When odometry quality is mysteriously bad, recalibrate before tuning.

The honest-evaluation corollary: every one of these degrades accuracy long before it triggers a failure signal. Estimator health flags are optimistic; ground truth (motion capture, or simulation — the B2 spine’s advantage) is the only arbiter that does not lie.

Evaluating an estimator

Standard metrics, all computed against ground truth after aligning trajectories (Umeyama alignment; yaw-and-position only for VIO, since roll, pitch, and scale are observable and should not be aligned away):

  • ATE (absolute trajectory error): RMSE (root-mean-square error) of position over the aligned trajectory. One number, dominated by drift; the headline benchmark metric.
  • RPE (relative pose error): error accumulated over fixed sub-trajectory lengths (e.g. per 10 m). Measures local odometry quality independent of loop closure, and is the honest number for a drifting system.
  • Percent drift (error per distance traveled), and for mapping, point-to-plane or chamfer distance of the reconstructed surface — the same family of metrics GLEAM-Bench uses for coverage evaluation.

evo is the standard tool; it consumes TUM/EuRoC/KITTI trajectory formats and does alignment and plotting.

Benchmarks that matter: EuRoC MAV (the VIO standard — onboard drone stereo+IMU with mm ground truth), TUM-VI (handheld fisheye, harder lighting), KITTI (outdoor automotive), and the Hilti SLAM Challenge (construction sites — the closest public proxy for hostile indoor capture conditions).

Simulation adds the one thing real benchmarks cannot: perfect, free ground truth for every run plus controllable negative controls (inject pose error, break the IMU model, remove texture). That is the evaluation posture of the B2 block of our Isaac lesson plan — measure drift against ground truth in sim, so that on hardware, where truth is unavailable, the failure signatures are already familiar.

Relation to SfM and photogrammetry

Offline reconstruction — structure-from-motion (SfM) as implemented by COLMAP — is the same mathematics without the real-time constraint: match features across all images, solve one global bundle adjustment over every pose and point. Because it sees everything at once, it beats any online system on accuracy; because it is batch, it can take hours and can fail wholesale on sequences an online tracker would have survived incrementally.

How the two relate in a capture pipeline:

  • SLAM poses initialize SfM. COLMAP’s hardest and slowest stage is figuring out approximate poses from nothing; online odometry provides them nearly free, turning global SfM into a refinement problem. ARKit/ARCore poses accompanying phone captures serve the same role (the pose-capture thread in 3D photo reconstruction).
  • SfM output feeds novel-view synthesis. 3D Gaussian Splatting and NeRF (neural radiance field) pipelines consume COLMAP poses/points as their standard input format — the reconstruction quality ceiling is set by pose quality.
  • Online SLAM guarantees coverage; offline SfM delivers quality. The drone flies on VIO, the coverage planner decides completeness from the dense map, and the high-quality imagery is reconstructed offline afterwards. The final model does not inherit online drift — SfM re-solves geometry from scratch, using online poses only as hints.

This split — navigate on the fast rough estimate, reconstruct on the slow accurate one — is the architecture assumption of the whole indoor capture project.

Fit to the indoor capture project

Where each piece lands for us:

  • Odometry is a solved procurement problem, not a research problem. Every candidate platform ships it: qVIO on VOXL, cuVSLAM on Jetson, ARKit/ARCore on phones. We should not write a VIO system; we should characterize the one we get (drift rate, failure conditions, health-signal honesty) and design around it.
  • Loop closure is the open question per platform. qVIO has none — on VOXL, map consistency over a whole-house mission is our problem, solved either by mission structure (short submaps, frequent revisits of a start region) or by running an external SLAM layer (OpenVINS/VINS-Fusion on the companion side) alongside the shipped VIO.
  • The dense map is where our product logic lives. Occupied/free/unknown drives frontier exploration and the coverage ledger; TSDF/ESDF drives safe planning. voxblox (on-platform) and nvblox (PC-side) are the defaults — this layer is open, mature, and ours to orchestrate rather than reimplement.
  • The two-frame discipline is a design requirement from day one. Smooth odom for control, consistent map for planning, corrections as transforms, jumps as planning events. Retrofitting this after building on a single-frame assumption is a rewrite.
  • Sim-first evaluation. Measure estimator drift and map corruption against ground truth in Isaac/Pegasus (B2), with injected-error negative controls, before trusting any of it on hardware where no ground truth exists.

The competitive read from the landscape survey holds here: estimation and local mapping are commodities; autonomous goal selection on top of them — where to go so the capture is complete — is the differentiating layer, and no shipped stack provides it.

References

Surveys and foundations:

Code and docs:

Related notes: VOXL 2 Mini for the shipped qVIO/voxblox stack, the Pixhawk ecosystem for the flight-stack side of the interface, ROS, from zero for tf2/REP-105 machinery, and autonomous drone navigation for what gets built on top.