LEt’s start by presenting the end result.

Matterport — this is our target when it comes to image stitching / navigation:

We can build on existing Open-soirce software, like Mrzipano. There are a few existing projects that have very nice interactive 3D panoramas:

Marzipano hotspot styles demo (Marzipano, Apache 2.0 - permissive - don’t have to distribute the code) Pannellum (MIT - very permissive, can do whatever)

Common limitation - no “live” navigation between panoramas. Can click on a marker and teleport to another panorama, but not smoothe transition.

Our job - take one of the exsting implementations, implement the transitions between panoramas. This is not a trivial task: - Deriving pose (camera position in 3d space) from just photos. - ARKit (Apple) / ARCore (Android/Apple) - capture with the phone, metadata already includes the estimates for position/orientation. Both use SLAM / SfT internally, so they are not magic bullet. - If we don’t use ARKit/ArCore, we need to implement fill SLAM / StF ourselves - extrac location/position/scene geometry from photos only. - If we want to use a 360 camera, mount phone rigidly together with camera. Capture metadata+photos with phone, 360 photos with the camera, sync in post. - Need to bench ARKit vs ARCore, is one significantly better than the other? - Scane editor. Need to combine the scene photos + metadata into a scena graph based on the room layout map. Capture dump -> proposed scene graph -> human fixes -> published tour The editor will allow you to adjust the node position, pick active nodes and navigation paths. - Viewer - takes the pre-processed data, makes it possible to navigate around the house, click on info markers.

Alternative approach - Gaussian splat instead of series of panoramas. Allows complete freedom of movement, which complicated things, but looks amazing.

Navigation

Cool things to show:

Antonomous discovery:

Next Best Path algorithm demo: > NextBestPath (NBP) is a novel method for next-best-path planning in 3D scene exploration. Unlike previous methods, NBP is designed to directly maximize mapping efficiency and coverage along the camera trajectory.

GLEAM (Learning Generalizable Exploration Policy for Active Mapping in Complex 3D Indoor Scenes):

But keep in mind, this is academic work. It will require an enormous effort to adapt.

This does not take into the account any of the complexity of flying a real drone inside a real house, real-time reconstruction of the 3d environment, etc, etc.

The environment to use for training seems to be https://developer.nvidia.com/isaac/sim