Can we use localization with a map generated by other camera?

[Shunmo17]

I would like to use localization with a smartphone and a map generated by mapping with RICOH Theta V, but it doesn't work at all when I tried.

Is OpenVSLAM can be used localization with map generated by the other camera? If yes, I would like to ask how to use it.

[ymd-stella]

See xdspacelab/openvslam#388.

[Shunmo17]

When I input the THETA image as a video with one frame for localization, I got the below result in which the some feature points were red. Also, it seems to estimate the position and posture correctly.

However, when I input the monocular camera image for localization, I got the below one in which no feature points were red.

[ymd-stella]

It's hard to pinpoint a single cause, but there are a few things that concern me.

• The distortion parameter of the camera is too large
• Use video that includes images from the same viewpoint and angle as those taken by THETA
• Run it with the --debug option to see why relocalization fails

[Shunmo17]

Thank your for your reply.

The distortion parameter of the camera is too large

I calibrated it again, and I got the below parameters.
I think they are correct, because I can get distortion corrected image with using them.

  fx: 547.65828438019
  fy: 548.401022528994
  cx: 332.617906440157
  cy: 260.026437608394

  k1: 0.177720142446549
  k2: -1.00036844855803
  p1: 0.0124202226453015
  p2: -0.00346415752137859
  k3: 1.09287028843021

Use video that includes images from the same viewpoint and angle as those taken by THETA

I have already used the same viewpoint and angle image for localization.

Run it with the --debug option to see why relocalization fails

I run localization with the below command, and I got the result.

${OPENVSLAM_PATH}/openvslam/build/run_video_localization \
        -v ${OPENVSLAM_PATH}/openvslam/build/orb_vocab.fbow \
        -m ${VIDEO_PATH} \
        -c /openvslam/config/iPhoneX_photo.yaml \
        --frame-skip 1 \
        --no-sleep \
        --map-db /openvslam/data/map.msg \
        --eval-log \
        --debug \

OpenVSLAM root:/openvslam$ ./run_localization.sh data/20210918_smartphone/iPhoneX/localization_resized_for_theta/IMG_9388.jpg 
input video: data/20210918_smartphone/iPhoneX/localization_resized_for_theta/IMG_9388.jpg
[2021-10-02 12:04:30.362] [D] CONSTRUCT: config
[2021-10-02 12:04:30.362] [I] config file loaded: /openvslam/config/iPhoneX_photo.yaml
[2021-10-02 12:04:30.362] [D] load camera model type
[2021-10-02 12:04:30.362] [D] load camera model parameters
[2021-10-02 12:04:30.362] [D] CONSTRUCT: camera::base
[2021-10-02 12:04:30.362] [D] CONSTRUCT: camera::perspective
[2021-10-02 12:04:30.362] [D] compute image bounds
[2021-10-02 12:04:30.362] [D] CONSTRUCT: system
[2021-10-02 12:04:30.362] [I] 
  ___               __   _____ _      _   __  __ 
 / _ \ _ __  ___ _ _\ \ / / __| |    /_\ |  \/  |
| (_) | '_ \/ -_) ' \\ V /\__ \ |__ / _ \| |\/| |
 \___/| .__/\___|_||_|\_/ |___/____/_/ \_\_|  |_|
      |_|                                        

Copyright (C) 2019,
National Institute of Advanced Industrial Science and Technology (AIST)
All rights reserved.

This is free software,
and you are welcome to redistribute it under certain conditions.
See the LICENSE file.

Camera:
  cy: 260.026437608394
  k1: 0.177720142446549
  k2: -1.00036844855803
  setup: !<!> monocular
  cx: 332.617906440157
  name: !<!> EuRoC monocular
  fx: 547.65828438019
  fy: 548.401022528994
  model: !<!> perspective
  p1: 0.0124202226453015
  p2: -0.00346415752137859
  k3: 1.09287028843021
  cols: 672
  rows: 504
  fps: 30.0
  color_order: !<!> RGB
Tracking:
  max_num_keypoints: 2000
  ini_max_num_keypoints: 4000
Feature:
  num_levels: 8
  scale_factor: 1.2
  min_fast_threshold: 7
  ini_fast_threshold: 20
Mapping:
  baseline_dist_thr_ratio: 0.02
PangolinViewer:
  keyframe_size: 0.07
  camera_line_width: 3
  graph_line_width: 1
  viewpoint_x: 0
  camera_size: 0.08
  point_size: 2
  keyframe_line_width: 1
  viewpoint_y: -0.65
  viewpoint_z: -1.9
  viewpoint_f: 400

[2021-10-02 12:04:30.363] [I] loading ORB vocabulary: /root/catkin_ws/src/openvslam/build/orb_vocab.fbow
[2021-10-02 12:04:30.377] [D] CONSTRUCT: data::camera_database
[2021-10-02 12:04:30.377] [D] CONSTRUCT: data::map_database
[2021-10-02 12:04:30.377] [D] CONSTRUCT: data::bow_database
[2021-10-02 12:04:30.378] [D] CONSTRUCT: publish::frame_publisher
[2021-10-02 12:04:30.378] [D] CONSTRUCT: publish::map_publisher
[2021-10-02 12:04:30.378] [D] load depth threshold
[2021-10-02 12:04:30.378] [D] load depthmap factor
[2021-10-02 12:04:30.378] [D] load maximum distance threshold where close keyframes could be found
[2021-10-02 12:04:30.378] [D] load maximum angle threshold between given pose and close keyframes
[2021-10-02 12:04:30.378] [D] CONSTRUCT: module::initializer
[2021-10-02 12:04:30.378] [D] CONSTRUCT: module::relocalizer
[2021-10-02 12:04:30.378] [D] CONSTRUCT: tracking_module
[2021-10-02 12:04:30.378] [D] load ORB parameters
[2021-10-02 12:04:30.378] [D] CONSTRUCT: mapping_module
[2021-10-02 12:04:30.378] [D] load mapping parameters
[2021-10-02 12:04:30.378] [D] load monocular mappping parameters
[2021-10-02 12:04:30.378] [D] Use baseline_dist_thr_ratio: 0.02
[2021-10-02 12:04:30.378] [D] CONSTRUCT: global_optimization_module
[2021-10-02 12:04:30.378] [I] clear map database
[2021-10-02 12:04:30.378] [I] clear BoW database
[2021-10-02 12:04:30.378] [I] load the MessagePack file of database from /openvslam/data/map.msg
[2021-10-02 12:04:32.609] [I] decoding 1 camera(s) to load
[2021-10-02 12:04:32.609] [I] load a camera "Insta360 One X2" from JSON
[2021-10-02 12:04:32.609] [D] CONSTRUCT: camera::base
[2021-10-02 12:04:32.609] [D] CONSTRUCT: camera::equirectangular
[2021-10-02 12:04:32.609] [D] compute image bounds
[2021-10-02 12:04:32.771] [I] decoding 114 keyframes to load
[2021-10-02 12:04:33.826] [I] decoding 12054 landmarks to load
[2021-10-02 12:04:33.834] [I] registering essential graph
[2021-10-02 12:04:34.085] [I] registering keyframe-landmark association
[2021-10-02 12:04:34.376] [I] updating covisibility graph
[2021-10-02 12:04:34.445] [I] updating landmark geometry
[2021-10-02 12:04:34.849] [I] startup SLAM system
[2021-10-02 12:04:34.850] [I] start mapping module
[2021-10-02 12:04:34.850] [I] start global optimization module
[2021-10-02 12:04:34.855] [I] pause mapping module

[ymd-stella]

If you are using the latest version of OpenVSLAM, and if you are inputting the images properly, then OpenVSLAM is searching the BoW database for candidates and not finding any. Try adding a debug log to the following function to see where the problem is.
https://github.com/OpenVSLAM-Community/openvslam/blob/1b30f1b1d46b5b2dff3d46fc2f1a2cdfba8c6ebf/src/openvslam/data/bow_database.cc#L127-L185

[Shunmo17]

Thank you, I will check the function.