荘の学習メモ

CUT3R for Egomotion Estimation: Advantages over Traditional Visual Odometry Executive Summary This comprehensive analysis examines CUT3R (Continuous Updating Transformer for 3D Reconstruction), a revolutionary approach to egomotion estimation that significantly outperforms traditional visual odometry methods. Through continuous state updates, dense 3D representation, and metric scale recovery, CUT3R addresses fundamental limitations of conventional approaches while enabling new capabilities in autonomous navigation, robotics, and augmented reality applications. Table of Contents 1. Introduction 2. Understanding CUT3R 2.1 Core Architecture 2.2 Key Technical Features 3. Traditional Visual Odometry Overview 3.1 Pipeline and Limitations 4. Detailed Technical Comparison 5. Performance Analysis 6. Practical Applications 7. Future Implications 8...

1. Install tiscamera.deb 2. Install dependencies as Dependencies link or as follows: git clone https://github.com/TheImagingSource/tiscamera.git cd tiscamera # Build dependencies sudo ./scripts/install-dependencies.sh --compilation --runtime # Runtime dependencies sudo ./scripts/install-dependencies.sh --runtime mkdir build cd build cmake -DBUILD_ARAVIS=OFF -DBUILD_GST_1_0=ON -DBUILD_TOOLS=ON -DBUILD_V4L2=ON -DCMAKE_INSTALL_PREFIX=/usr .. make sudo make install Test: 1. gst-launch-1.0 tcambin ! videoconvert ! ximagesink 2. tcam-capture *Needed library and head files: [4] Reference: [1]  https://github.com/TheImagingSource/tiscamera [2]  https://www.argocorp.com/cam/ImagingSource/Linux.html [3]  https://github.com/TheImagingSource/Linux-tiscamera-Programming-Samples [4]  https://github.com/TheImagingSource/tiscamera/blob/master/examples/c/Makefile

Removing OpenCV installed by JetPack before performing this script installation: $ sudo apt-get purge libopencv* [Step by step commands] $ cd ~ $ wget -O opencv.zip https://github.com/Itseez/opencv/archive/ 3.3.1 .zip $ unzip opencv.zip $ cd opencv $ wget -O opencv_contrib.zip https://github.com/Itseez/opencv_contrib/archive/ 3.3.1 .zip $ unzip opencv_contrib.zip $ mkdir build $ cd build $ cmake -D CMAKE_BUILD_TYPE=RELEASE                 -D CMAKE_INSTALL_PREFIX= /usr/local                  - D WITH_CUDA=ON                  -D ENABLE_FAST_MATH=1                 -D CUDA_FAST_MATH=1                 -D WITH_CUBLAS=1                 -D INSTALL_PYTHON_EXAMPLES=OFF          ...

1. Install libharu (1) Download source files [1] (2) Install [2] tar -xvzf libharu-X.X.X.tar.gz  cd libharu-X.X.X mkdir build cmake ../ make sudo make install make clean (3) Download chessboard creation file [3] git clone https://github.com/cgdsss/chessboard.git cd chessboard cd src cmake ../ make (4) Run the executable file ./chesssboard [grid_size(mm)] [length_grid_num] [width_grid_num]                                          (ex: sudo ./chessboard 17 11 8 ) Reference: [1]  http://libharu.org/ [2]  https://github.com/libharu/libharu/wiki/Installation [3]  https://github.com/cgdsss/chessboard

1. Download "IDS Software Suite 4.90.06 for Linux" and install [1] 2. Start the uEye daemon by typing sudo /etc/init.d/ueyeethdrc start 3. Adjust lens using demo application ( /usr/local/share/ueye/bin ) 4. Download samples [2] 5. Build a project in nsight editor include: /usr/inlude/ueye.h library: /usr/lib/libueye_api.so Reference: [1] https://en.ids-imaging.com/tl_files/downloads/uEye_SDK/readme/ueye-linux-readme-49006_EN.html#installation [2] https://en.ids-imaging.com/programming-examples.html

記錄從raw image到depth map的工具及步驟。 (目的: 得到像直接使用ZED Camera SDK一樣的校正後影像及深度圖) [MATLAB (Stereo camera calibration)] 事先準備好左右相機的各角度拍攝校正板之影像，輸入APP即可得到Camera Matrix。 Reference:  Image Acquisition Toolbox  (可用來開啟XIMEA Camera) ステレオ キャリブレーション アプリ stereoCameraCalibrator Camera Calibration Toolbox for Matlab [OpenCV] Step1: Capture and save raw L/R images with chessboard in a folder. Step2: Calibration. Reference: Capture image , (1) StereoCapture.cpp Save image stereocalibrate.cpp (2) zed-opencv-native [Sum up of calibrating a XIMEA stereo camera] 1. Capture and save raw L/R images with chessboard in a folder. (nsight: Camera_realtime project) 2. Open Matlab input: stereoCameraCalibrator 3. Choose L/R images path and chessboard width (ex:23mm) 4. Click calibrate 5. Click Export camera parameter -> generate MATLAB script *Command: " cameraCalibrator" for single camera calibration

最重要:  SLAM 完整學習資源 Triangulation 1.  OpenCVの三角測量関数『cv::triangulatepoints』 2.  slambook/ch7/triangulation.cpp Depth map 1.  disparity-map 2.  lsd-slam源码解读第五篇:DepthEstimation 3.  StereoBM 4.  Stereo_Matcher Pnp 1.  slambook/ch7/pose_estimation_3d2d.cpp SFM 1..  Robotic Vision : Sensors, Localization and Control 2.  ORB-SLAM代码详解之SLAM系统初始化 3.  SFM三维重建源码（Matlab）

1. Initial setting:  Install OpenCV extra modules  Setting nsight environment (include and lib path)  Confirm ZED Serial number (/usr/local/zed/tools/ ZED Explorer )  Key in ZED SN to get camera matrix and rectified images 後來改成用ZED SDK得到camera matrix (每次run程式時，回傳值都有一點誤差，可能是環境溫度等等造成)  Open ZED Camera and get real-time rectified images Reference: Homepage zed-examples zed&opencv zed-opencv-native 2. Feature matching / Stereo matching: Convert ZED source images to grayscale images Declare an OpenCV matcher (ex: BFMatcher )  Search match points between two images (left and right) Storing keypoints into vectors  Using RANSAC method to find good stereo matching points ( findHomography OpenCV Fundamental matrix function ) Draw and Display good stereo matching points according  RANSAC_mask   Reference: 立体匹配算法（Stereo Matching）及其在OpenCV中的应用 OpenCV Stereo Matching の性能比較 Viulib tutorial_feature_flann_matche...

[Install OpenCV's extra modules] Key: OpenCV and OpenCV contrib's version should be the same. (ex:3.3.1) (1) Download OpenCV 3.3.1 and OpenCV contrib 3.3.1 (Even OpenCV has been installed through Jetpack) (2) Put OpenCV contrib 3.3.1 folder into OpenCV 3.3.1 folder (3) Create a build folder and cd to the directory (4) Command: $ cmake -DOPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules <opencv_source_directory> $ make -j5 (5) Open cmake GUI (cmake-gui) (6) Follow the step2-7, extra source files and libraries should be correctly installed at /usr/local/include & /usr/local/lib Reference:  opencv_contrib ,  安裝OpenCV + OpenCV_Contrib ,  jetsonhacks

Concept: 隨機選擇一組資料的子集，計算這個子集的轉換式，找出符合此轉換式的所有資料，稱之為inliers，反覆隨機選取子集直到inliers數量多到足以代表整組資料時。 RANSAC Line fitting example: RANSAC computer vision applications: RANSAC for feature matching RANSAC for homography Reference: 交大開放式課程: 機器人視覺 (單元9 Sensor Data, RANSAC, FORVIS )

1. Feature tracking: Harris + KLT (一般常用的組合),  SIFT (比較穩，但計算時間也較長) , Controlled environment (可確保特徵點匹配準確) Challenges: Removing false corners (需濾掉匹配錯誤的特徵點), assuming stationary background (物體靜止較方便相機姿態估算), deleting short tracks (盡量用會一直出現的特徵點), growing matches (盡可能增加穩健特徵點數量). 2. Essential matrix: 當已知相機參數時，Fundamental matrix可簡化為Essential matrix。對Essential matrix進行SVD分解可進一步求得兩相機座標系統之間的R和T (假設兩張圖由不同位置之相機所拍攝)。 解法參:   淺談基礎矩陣，本質矩陣與相機移動 3. Singular value decomposition (SVD): 用於解一個非對稱矩陣的eigenvalues, eigenvectors. 4. Triangulation: 探討當已知兩張從不同角度拍攝的物體影像，如何建構出3D物體資訊。 理論上: 簡單數學幾何關係即可找出物體在3D中的位置 實際上: 兩個相機不會剛好平行 需要在影像上找到確定地相對特徵點 解決方法: 相機事先校正 在epipolar line上找對應特徵點 (Sum of Squared Differences (SSD) algorithm, Energy minimization algorithm) Image rectification使相對應特徵點平行 OpenCV function: trinagulatePoints (參:  triangulate 3d points from a stereo camera and chessboard. ) 實際雙眼視覺reconstruction流程: Calibrate cameras (得到相機參數) Rectify images (對齊特徵點) Com...

1. Epipolar Geometry Two images taken at the same time, in different positions, by different cameras. Don't need to search across the entire image to estimate the motion vector Only one degree of freedom for the possible correspondence 假設有兩張連續拍攝的影像 I1, I2 ，則物體 P 在 I2 上的成像點必落在一條線上，稱之為Epipolar line(因為不知道P與 I1 的距離)，而 I2 上所有的 Epipolar line會相交在一點，稱之為epipole。 2. The fundamental matrix 在 I1及 I2 上相對應的特徵點，滿足: which is: We can build an linear system with n corresponding feature points located on two images: 找到最佳解f即為Fundamental matrix。 Basic algorithm: normalized 8-points algorithm 重要觀念： Normalized特徵點。 雖然無法找到唯一解使得Af = 0，但可對A做SVD特異值分解，D之中最小eigenvalue對應的eigenvector即為使Af最小的向量。(Af = 0 => Af = λf = 0, satisfied when λ=0 ) 到Step4算出來的F可能不是Rank2，所以需要再做一次SVD，令D之中最小eigenvalue為0，使F變為Rank2。 RANSAC可用來濾掉outliers。 Image rectification可用來對齊兩張影像的epipolar line，方便我們搜尋特徵點。(homography transformation) 參: Hartley.

Visual Odometry: The process of incrementally estimating your position and orientation with respect to an initial reference frames by tracking visual features. Refers to estimate not only the distance traveled, but the entire trajectory of a moving robot . Monocular Visual Odometry & Stereo Visual Odometry. (Stereo case is usually much more robust and concrete but restricted by the distance between objects. ) Difference between visual SLAM (Simultaneous Localization and Mapping): Visual SLAM: produces also map of features  Visual odometry: focuses on the camera trajectory. Steps (Stereo): Key: Perfectly feature point matching Reference: Visual Odmetry from scratch - A tutorial for beginners Monocular Visual Odometry using OpenCV Introduction to VO Real-time Stereo VO Multiple View Geometry Data base for testing algorithm Source code: libfovis ( Visual odometry library ) Think: 1. How to shorten the processing time ? (Pa...