• Hopefully we can get most bugs and doc issues done before and June can be more focused on unit and install tests.

Victor

• Suggest next directions for Ilya, Maria and yourself for the May through summer period. Some options (in random order):
  • Tutorials/OpenCV Class/Docs
    • It is very worthwhile to spend time on this and I don’t think we can trust GSoC students to do the whole thing
  • Object Recognition
    • Hinterstoisser’s new paper (Line-Mod) on binary codes, combining something like DOT with Depth. Needs to be made
      • Modular so new modalities can be added and/or used independently
      • These templates are not complete, just indexes into where features occured withing an ROI but they do naturally subdivide into parts. We want to use the learning of parts to approach occlusion.
    • Cloth, deformable objects — open area we have nothing on so far
    • Transparent — I will be approaching this and it might use a hardware solution of a light that can flash on and off for specularity
    • articulated objects
• People
  • Torso, upper torso, arm, hand, head — knowing where these are in 3D
• High value classifiers
  • This is the same as “People” but maybe for other objects like cups, plates, silverware … chairs tables etc
  • Just like we have a popular face detector, do this for other objects

Kurt

Ethan

James

From last time

Gary

• Send Victor, Vadim Ecto
• Update Victor on latest CVFH

Radu

Alexander

Vadim

Victor

Kurt

Ethan

James

Agenda

• GSoC final mentor+project allocation due!

Minutes

• VSLAM Some configurations get a crash in SBA with forward pose (3D to 3D only). SVD based works
  • Removing dependencies … need Kurt’s advice
• TOD
  • Test system doesn’t work with latest changes in TOD detecting
  • 2 version,
    • Vincent with ORB Good on 1Mpix, some issue at 5Mpix
    • Alexander better on 5Mpix, worse on kinect with SURF and SURF
• DOT Ilya working on fitting pose based on edges, only those that are invariant to viewpoint
  • Optimizing Haussdorf distance … can have problems in clutter. But Hausdorf distance is large
  • Is pose failure used to filter?
• 6DOF head pose
  • Have a student looking at using poser to do this with kinect data
• Better background subtraction added by Zoran Zivkovic and a patch by Konstantin Krivakin
  • This works quite well “out of the box” and detects shadow.
• Input and output arrays
  • cvADD(inputM1,inputM2,outputM); Previously we used arrays, now can use STL vectors etc
  • Possible to allow OpenCV to process eigen matrices
• GSoC
  • Tutorials
  • 2 for iOS
  • Good tutorial on using Sphinx
  • Should link theory to a book/Learning OpenCV? Or course outline
    

    
    
    12 slots:
    
    2 for Vincent: SfM Mechanical Turk
    
    Motion tracking: Serge [we have a good student for this and interested mentor]
    
    Object tracking: Alexander [we need to coordinate this with Serge]
    
    2 iOS Port: Victor
    
    2 OpenCL Interface: Vadim
    
    OpenCV samples in python: Vadim
    
    2 Tutorials and Examples: Gary [I could see 2 slots for this … I could use help]
    
    Kurt — VSLAM into OpenCV [  Use NVidia team instead
    
    Peter — I don’t remember what your interest this round was. There is an interesting gradient domain (aka blending) proposal
    
    Caroline — Human rec/tracking. I can also see value in “high value classifiers”, that is, just taking something like the rejection cascade (which now can use any feature but has Haar and LBP available) and getting it to work well on different views of the head, eyes, or any body part.
    
    

Victor

TOD: Updating test system to work with the latest changes in tod_detecting:
- fixed bug in TODRecognizer with clustrering (50898 rev)
- modified test system for tod_* (50973 rev)
- implemented infrastructure for resizing train and test bases in tod_* format (51021, 51022 rev)
- fixed #5097 issue (50974 rev)
- answered some questions from Dejan [Alexander]

Untextured objects: experimented with pose refinement algorithm on 'plates, cups and bowls' dataset. The algorithm does not give good results the most of the time because it uses object edges that don't change with a viewpoint. As it appears, uniformly colored kitchen items have few such edges and they are weak. We are working on an algorithm that uses test point cloud to do detection and pose estimation, using the same approach (global optimization from nlopt package). This algoritm performs well on a single object (see attachments -- initial.png with initial pose and final.png with pose after optimization) but fails in clutter. [Ilya]

Implemented a function for calculating a mask of train objects using point clouds. The main reason is that the existing algorithm in tod_training depends on R,T found using template and often this is not accurate enough, sometimes an object on a table can have has negative z (formally being under the table), resulting in heavy clipping. Our function computes the mask in camera space (independent from R,T quality) and is based on fitting a plane to a table. It gives more accurate masks for all objects from our "plates_cups_bowls" base if object is taller than ~1.5 cm. [*Maria*]

Buildbot:
- added builders for MacOs
- added run tests for Python's module
- tested authorization scripts
- begin to work on LogObserver class for python tests results
- tried to compile OpenCV with CLang on MacOS
- we need to compile python from sources in Windows to run python wrappers in Debug mode. This is in progress. [Andrey, Alexander]

Tutorials: work in progress on an extensive tutorial of projective geometry with opencv (see the contents below). The foils are 90% done. [Victor]
Contents (largely based on 'multiple view geometry' and Golub for theory review):
Single-view geometry
Pinhole camera model
PnP problem, cost functions (reprojection error)
Direct Linear Transformation, RANSAC, LM
Camera calibration
Planar objects, homography
Two-view geometry
Epipolar geometry, fundamental matrix, Essential matrix
Structure from motion
3D reconstruction

• 
• Initial fit
• 
• Final fit

Vadim

# (User Contributions) integrated improved mixture-of-gaussian background subtraction algorithm by Zoran Zivkovic (this is incremental improvement of the previous version, contributed by the same author); integrated another patch by Konstantin Krivakin - for parallel TBB-based SVM prediction.

#  (Documentation)
Fixed most issues reported by tech writer. Some of the issues required fixing Sphinx. The patch has been posted to the Sphinx bug tracker.

#  (OpenCV interface) Finally, finished transition to InputArray/OutputArray. Fixed all the build problems and most of the problems in tests found by our continuous integration system.

plans.
# start cleaning bug tracker.
#  GSoC: define who will be accepted

Anatoly Baksheev

Accomplishments:

1)      Finished extracting Visual odometry package with GPU support from ROS. Now we need to decide how it should be integrated into OpenCV. Details were send by Victor yesterday.
2)      Image stitching
·         Spent a lot of time studying Structure from Motion theory (sparse bundle adjustment, starting point computation for LM, papers about SBA on GPU, decomposition homography into rotation and translations, etc.)
·         We decided to stop on RotationCameraModel and finish stitching functionality for spherical, cylindrical and planar panoramas. Currently we are working on final (not research) code.
We can consider two models of camera movement: 1) camera only rotates 2) also shifts. In the first case here is quite simple model and we are going to finish it this week. For second case, there is a quite complex algorithms and a lot of things to implement. We decided not to do this. It more close to Structure From Motion (SfM) area rather than panorama composing.  And it’s implementation may take very big amount of time.
3)      Started work on slides for GTC Israel.
Plans:

1)      Finish synthesis of panorama for RotationCameraModel.
2)      Optimization  of GPU stitching stuff.
3)      VSLAM into OpenCV
4)      New tasks??
Issues:

·         Optimal seam selection will use CPU graphcut due to instability of GC from NPP.

Action Items

Gary

Radu

Alexander

Vadim

Victor

• Talk to Kurt about directing someone on the NVidia team for VSLAM

Kurt

Ethan

James

From last time

Gary

Radu

Alexander

Vadim

• Figure out a good time for a “bug hackathon” week and a “doc-athon week”.
  • I will be out May 7-17 but can maybe do some work while traveling
• Figure out a target date for the next release. Probably in June.
  • Hopefully we can get most bugs and doc issues done before and June can be more focused on unit and install tests.

Victor

• Suggest next directions for Ilya, Maria and yourself for the May through summer period. Some options (in random order):
  • Tutorials/OpenCV Class/Docs
    • It is very worthwhile to spend time on this and I don’t think we can trust GSoC students to do the whole thing
  • Object Recognition
    • Hinterstoisser’s new paper (Line-Mod) on binary codes, combining something like DOT with Depth. Needs to be made
      • Modular so new modalities can be added and/or used independently
      • These templates are not complete, just indexes into where features occured withing an ROI but they do naturally subdivide into parts. We want to use the learning of parts to approach occlusion.
    • Cloth, deformable objects — open area we have nothing on so far
    • Transparent — I will be approaching this and it might use a hardware solution of a light that can flash on and off for specularity
    • articulated objects
• People
  • Torso, upper torso, arm, hand, head — knowing where these are in 3D
• High value classifiers
  • This is the same as “People” but maybe for other objects like cups, plates, silverware … chairs tables etc
  • Just like we have a popular face detector, do this for other objects

Kurt

Ethan

James

Agenda

• GSoC

Minutes

• Tutorials
  • Write topics
  • Sample of how to solve a meaningful problem
  • Line by line explanation of how to solve
  • Victor has checked in a tutorial to trunk under
    • opencv/doc/opencv_tutorials.pdf
    • opencv/doc/tutorials with the latex sources
• Set up payment for the org

Vadim

# (New features) Lapack is not used in OpenCV anymore. The only thing why we needed it was SVD, which now has been replaced with very robust and compact hand-written Jacobi SVD. On large matrices (like 100x100) Jacobi algorithm can be 50-100% slower than Lapack, but on small matrices (3x3 - 10x10) it is several times faster. It's planned to add Eigen3 support by # 3 release time.

#  (Bug tracker) 32 tickets have been closed. Some analysis of the open bugs has been done:

problem domain       critical/major/minor/patches

build/install           0/12/12/2
documentation     43 in total
core                     0/10/5/1
highgui                 4/30/23/17
imgproc+video      0/18/17/6
python binding      4/10/9/2

all                       9/125/63/35

plans.
# continue cleaning bug tracker.

Victor

tod: added support for resizing datasets in new (kinect) format. Now we can convert prosilica datasets to the new format and change image resolution. Together with Dejan fixed issue #5097 issue. Updated tod rosinstall file. Got a dependence of detection and false positive rates on test images resolution (with training images resolution fixed at 5mp). The dependence is shown below, one can see that detection rate drops close to zero for two objects and stays on sufficiently high level for another two. High false positive rate for 4medals at 5mp is a bug that is fixed in the latest revision of the code. [Alexander, Victor]

Howard 3d<->3d pose estimation: a lot of progress in understanding why vslam does not work with Howard. A crucial difference in between finding a solution by ransac and by clique. All other differences have been eliminated. It looks like the solution found by RANSAC produces inliers that are not consistent in Howard's sense with the thresholds that we selected. If we increase the threshold, we get too many outliers in the maximum clique and a worse solution that also produces bad visual odometry, even after sba. [Alexander]

Buildbot:
- modified scripts for MacOs (now we have some problems with running Python tests on MacOS in the case of  CMAKE_OSX_ARCHITECTURES=i386)
- finished LogObserver class for python tests results. Now a user can see number of failed and successful tests
- implemented scripts for automatic start of buildbot after reboot BigMac
- added sending e-mail after every failed build (temporary messages are set only to Alexander and Andrey)
- created issues (#1019, 1021, 1027)
- begin to implement scripts for testing Cuda model in OpenCV

Pose refinement: the code has been refactored and put to object recognition stack in the package edges_pose_refiner. [Ilya]

DOT: first version of DOT is in opencv trunk, still has to be tested and optimized. Tests on "plates_cups_bowls" dataset showed that the method works reliably for matted untextured objects that are different in color from background. [*Maria*]

Anatoly Baksheev

1)      OpenCV  & GPU
·         Added 16-bit support to TiffEncoder
·         Visual Odometry integration is frozen. Waiting decision how to integrate, and a VSLAM update from Kurt (Willow).
·         Review and refactoring of GPU Module. Going to use some Thrust functionality to make our code more standard, and create Thrust extension for pitched memory.
o   We reuse from Thrust or implement: functors, binders, micro algorithms, like plus, min, reduceSMEM<T, int>, our analogue of transform_iterator but for pitched memory, our analogue of zip_iterator.
Won’t commit it before support of Cuda# 0.
2)      Stitching
·         Created spherical, cylindrical, and plane warpers.
·         Refactored image stitching code – now it’s easy to switch blender (warper, seam finder) from one to another.
·         Rotation model is still in development. We can stich many images (some of results attached). But sometimes results are incorrect, or bad estimated parameters do not allow to stitch. Investigating.
Plans:

·         Finish image stitching application.
·         Meeting with Radu to start PCL Cuda project.

Action Items

Gary

• Look at tutorials and see if what we want to do
• Look at GSoC payment setup

Radu

Alexander

Vadim

Look at GSoC payment setup

• Tutorial suggestions

Victor

• Get me a list of tutorials suggestions

Kurt

Ethan

James

From last time

Gary

Radu

Alexander

Vadim

• Figure out a good time for a “bug hackathon” week and a “doc-athon week”.
  • I will be out May 7-17 but can maybe do some work while traveling
• Figure out a target date for the next release. Probably in June.
  • Hopefully we can get most bugs and doc issues done before and June can be more focused on unit and install tests.

Victor

• Talk to Kurt about directing someone on the NVidia team for VSLAM
• Suggest next directions for Ilya, Maria and yourself for the May through summer period. Some options (in random order):
  • Tutorials/OpenCV Class/Docs
    • It is very worthwhile to spend time on this and I don’t think we can trust GSoC students to do the whole thing
  • Object Recognition
    • Hinterstoisser’s new paper (Line-Mod) on binary codes, combining something like DOT with Depth. Needs to be made
      • Modular so new modalities can be added and/or used independently
      • These templates are not complete, just indexes into where features occured withing an ROI but they do naturally subdivide into parts. We want to use the learning of parts to approach occlusion.
    • Cloth, deformable objects — open area we have nothing on so far
    • Transparent — I will be approaching this and it might use a hardware solution of a light that can flash on and off for specularity
    • articulated objects
• People
  • Torso, upper torso, arm, hand, head — knowing where these are in 3D
• High value classifiers
  • This is the same as “People” but maybe for other objects like cups, plates, silverware … chairs tables etc
  • Just like we have a popular face detector, do this for other objects

Kurt

Ethan

James

Agenda

• GSoC
  • GSoC Payment
  • Tutorials
  • Other issues
• Release target date
• Further research
  • Size filtering
• Distance header

Minutes

• GSoC
  • Sergey is looking into gettting us signed up as suppliers
  • An admin might have to be involved, but that could be Vadim
• Tutorials
  • Do it in Sphinx
  • Vadim will set up the first “hello world tutorial”
• iPhone start tomorrow
• Release date/activities
  • Cleaning debug tracker
    • Integrating patches
    • Fixing major bugs
    • Maria already fixed most of the ML issues and Features2D bugs
    • Many HighGUI issues that are difficult since they are really ffmpeg issues that are environment dependent.
    • Marius on FLANN issues
  • Documentation: Tech writer reviewed most of the chapters
    • But not features2d (should be priority)
  • Would like to get ORB in the release since probably the ORB paper is accepted into ICCV and we’d like to announce it there with it already inside of OpenCV and usable for Features 2D.
  • Release timeline
    • Bug fixing period 3 weeks
    • Documentation 3 weeks
      • Code writer should look at the user wiki content
      • Search on documentation or documentation/cpp or …/c or …/py
    • ORB 1 week
    • Code coverage completion 2 No new functions without coverage
    • Testing 1 week
  • Aim for release by June 20th
• JAVA API for OpenCV
  • Tegra team Java binings by CVPR (June 20th)
• Distance header — Vadim thinks it can get in before the release

Vadim

# (Bug Tracker) 13 tickets have been closed.

#  (Build Issues) OpenCV now builds on Ubuntu 11.04 (after fix in highgui).

#  (Python) Python bindings for OpenCV 1.x API (by James) and OpenCV # x (auto-generated) have been split into separate modules: cv and cv2, respectively. That should resolve some name conflicts and simplify work on the Java bindings.

plans.
# continue to clean the bug tracker.

Victor

TOD:
- re-factored scripts for training base resizing for support Kinect bases (rev 51305)
- fixed bug in guess::draw method, when train and test bases have different camera parameters (rev 51293)
- fixed exception in CloudFilter, when bounding box for projected points has vertex with negative coordinates (rev 51293)
- correspondence with Dejan and Juilus. Begin to investigate their problems with tod_* (#5083, #5084, #5103, #5104). Learned Maria's code for cropping object for fixing ticket #510#  I plan to integrate her code to tod_training. [Alexander]

Pose refinement: doxygen documentation for the edges_pose_refiner package was created. All public methods and fields for external use are documented. Implemented code for running the edges_pose_refiner package with train and test data in the new format and run it on TOD poses. Implemented interpolation of point clouds to prosilica resolution (my old implementation used disparity images which are unavailable in the new data format). [Ilya]

OpenCV:
- fixed a bug in knnMatch (rev. 4969). I think it solves #5104 bug. I plan to check this.
- fixed a bug in solvePnPRansac test. Applied a patch of Pieter-Jan Busschaert for the code of solvePnPRansac  (rev. 4976). [Alexander]

- Added a new sample on MLL that classifies image points and shows a decision boundary (points_classifier.cpp). The base version of the sample was contributed by Alexander Bovyrin, I rewrote it in C++ interface of OpenCV and added 4 new models (CvNormalBayesClassifier, CvBoost, CvERTrees, CvGBTrees). So now it demonstrates all OpenCV supervised learning models as well as unsupervized CvEM. But CvGBTrees gives incorrect decision boundary, a report about this was sent to Pavel (the author of GBT). [*Maria*]

# ) Resolved the problems:
- tickets on features2d and ml modules: ##832, 977, 982, 978, 932, 899, 877, 887;
- fixed crashes in CvBoost (subsamle_mask) and CvGBtrees (C++ wrappers) discovered while adding points_classifier.cpp sample.

VisualSlam:
- Experimented with Hirschmuller constraints in finding the maximum clique. Similar problems are observed (we get too many outliers in the maximum clique and a worse solution that also produces bad visual odometry, even after sba). [Alexander]

Buildbot:
- added 2 builders for OpenCV in Ubuntu64 with WITH_CUDA=ON
- added builder for Android version of OpenCV on Ubuntu 64
- fixed different bugs in MacOs and Windows builders
- added creating OpenCV Linux tarball and HTML documentation every night
- added opencv ftp (thanks Andrew S.), we upload tarball and docs every time to ftp://argus-cv.dnsalias.org
Now we have two unsolved problems:
- incorrect uploading HTML pages with documentation to the FTP
- exception after running Python tests on MacOS, when we compile OpenCV for i386 architecture [Andrey, Alexander]

Other: May 1st was a public holiday in Russia.

Anatoly Baksheev

1)      OpenCV  & GPU
·         Modified CudaMem implementation for ZERO_COPY flag so that rows are aligned to 256 or 512 bytes. Hope after mapping to device memory this will be the same.
·         Fixed minor bugs in gpu::cvtColor.
2)      Stitching.
·         Finished with stitching algorithm for rotation camera model with global adjustment. Now we estimate initial parameters from pair-wise homographies. And next refine it using Lev-Marq. (i.e. solve not sparse Bundle Adjustment problem).
·         Improved stability of stitching (added filtering of bad estimated parameters such as negative focal lengths, etc. This caused crashes before).  Minor improvements in Voronoi and GraphCut seam finders.
Issues:

• GraphCut seam finding sometimes leads to artifacts.
• Parameters tweaking is required for some datasets.

Plans:
*Add stitching application to SVN tree, move some reusable parts of stitching to different OpenCV modules, add possibility to select CPU or GPU implementation (SURF, BFM, Blending, etc.).
*Test it more, people from Itseez will play with it.

3) Started work on PCL. First task will is to implement OctTree on CUDA/Thrust.

· Compiled PCL.

· Now we study PCL sources and papers about OctTree on GPU.

4) Started thinking about camera calibration without chessboard pattern. This will be our next task.

· GPU here can be used only for SURF and BFM. All other stuff will be on CPU.

Plans:
· Suggest calibration scheme and begin its implementation or at least do some experiments.

· Integrate stitching to OpenCV SVN tree and test it more.

· Work on porting PCL to Cuda.

• 
• Image stitching results

Action Items

Gary

• Contact GSoC interns
• Get foils to Victor

Radu

Alexander

Vadim

Create a tutorial stub

• Contact GSoC interns
• Make sure the tech writer incorporates the opencv.willowgarage.com/wiki/documentation user changes.

Victor

• Get me a list of tutorials suggestions
• Contact GSoC interns

Kurt

Ethan

James

From last time

Gary

• Look at tutorials and see if what we want to do
• Look at GSoC payment setup

Radu

Alexander

Vadim

• Figure out a good time for a “bug hackathon” week and a “doc-athon week”.
  • I will be out May 7-17 but can maybe do some work while traveling
• Figure out a target date for the next release. Probably in June.
  • Hopefully we can get most bugs and doc issues done before and June can be more focused on unit and install tests.
• Tutorial suggestions

Victor

• Look at GSoC payment setup
• Tutorial suggestions * Talk to Kurt about directing someone on the NVidia team for VSLAM
• Suggest next directions for Ilya, Maria and yourself for the May through summer period. Some options (in random order):
  • Tutorials/OpenCV Class/Docs
    • It is very worthwhile to spend time on this and I don’t think we can trust GSoC students to do the whole thing
  • Object Recognition
    • Hinterstoisser’s new paper (Line-Mod) on binary codes, combining something like DOT with Depth. Needs to be made
      • Modular so new modalities can be added and/or used independently
      • These templates are not complete, just indexes into where features occured withing an ROI but they do naturally subdivide into parts. We want to use the learning of parts to approach occlusion.
    • Cloth, deformable objects — open area we have nothing on so far
    • Transparent — I will be approaching this and it might use a hardware solution of a light that can flash on and off for specularity
    • articulated objects
• People
  • Torso, upper torso, arm, hand, head — knowing where these are in 3D
• High value classifiers
  • This is the same as “People” but maybe for other objects like cups, plates, silverware … chairs tables etc
  • Just like we have a popular face detector, do this for other objects

Kurt

Ethan

James

Agenda

• GSoC
  • GSoC Payment
  • Tutorials
  • Other issues
• Release target date
• Orb — Vincent has cleaned up
• New competition/direction

Minutes

• Image Stitching is now in OpenCV trunk, see 3rd photo below
• DOT from Hinterstoisser is now in trunk. It’s not SSE optimized yet and so runs at ~18ms/template
• Matrices in OpenCV are now indirect, using a proxy types for input. So inputs can be cv::Mat or std::vectors …
  • This is a step towards using Eigen out of the box
  • Unifying with PCL
• Solutions in perception
  • Winners in order were:
    

    
    
    Berkeley 68%
    
    Jacobs 66%
    
    Stanford 53%
    
    Baseline (TOD) 50% [But see below …]
    
    

  • TOD
    • TOD
    • Low resolution: Use FAST with low threshold, like 5 and non-max turned off works better
• Release:
  • Vadim is working on bugs and docs
  • We want the new feature, ORB in. Vincent will commit.
    • Docs should go …/modules/features2d/doc/common_interfaces_of_descriptor_extractors.rst
    • Tests should go to …/modules/features2d/test/test_features2d.cpp
      • Test data will go to opencv_extra: * /opencv_extra/testdata/cv/detectors_descriptors_evaluation/descriptors*
• Release date is still on for June 20th, right before CVPR

Victor

OpenCV: the problem in cv::DynamicAdaptedFeatureDetector (#5104 in ROS trac), modified constructors of cv::AdjusterAdapter's inheritors to pass min and max thresholds. [Maria, Alexander]
Fixed tickets ##1038, 1025, 1044, 1046, 1048. [*Maria*]

Buildbot:
- modified scripts for supporting Vadim's modifications (make docs, make html_docs)
- investigated problems with python failed tests on Windows and MacOS, reproduced these problems on two tests (PreliminaryTests.test_lena on MacOs, test_MixChannels on Windows32). Sent this information to Vadim. Found new problem: tests on MacOs are ignored OPENCV_TEST_DATA_PATH variable if WITH_TBB=ON. Plan to continue investigation of these problems.
- fixed problem with incorrect Python environment on Win64
- created a web-site with OpenCV documentation built daily by buildbot, it is available at opencv.itseez.com
- fixed scripts for the Android builder (this builder doesn't work after rebooting virtual machine)
- fixed Documentation builder after Vadim's modifications in docs folder (renaming folders, removing BUILD_REFMAN, ...)
- Created an external page http://pullrequest.opencv.org/ so that anybody can track opencv trunk status.  [Alexander]

TOD:
- finished re-factoring scripts for training base resizing, fixed bug when camera parameters file doesn't exist in the main training base folder,
fixed incorrect generating of pcds.txt file (rev 51320)
- added some useful parameters to frecognizer
-s - for saving images with projections into selected folder
-o - for sorting images in the image folder in the case of [1-9]*.png format
-h - for showing progress  (rev 51434)
- fixed bug in CloudFilter with incorrect projections (rev 51435)
- correspondence with Dejan and Juilus. I am investigating their problems with tod_*  (#5104 and #5103)
- #5104 I got from Julius their training and test bases, reproduced this problem. Maria and me localized this problem in DynamicAdaptedFeatureDetector, now we are trying to fix them.
- #510#  I begin to experiment with Maria's version of masker algorithm. I can get better masks on the one bag file with Kinect data (silk.bag) in comparison with default cloud masker in tod_training. I plan to do some more experiments and integrate it to the tod_training on this week. [Alexander]
- Fixed #5104 issue (OpenCV 5086 revision, tod_training 51452 revision)
- Integrated the new masker algorithm to the tod_training (#5103 issue). [Alexander, Maria]

DOT: current opencv implementation is 18.7ms per one frame and one training template, Hintestoisser's version with IPP and SSE -- 4ms. Optimization of opencv version is in progress. [*Maria*]

Pose refinement: edges_pose_refiner package has been tested together with tod. It can take a pose generated by tod as an input and refine it using edges cost function. An example is attached, it uses binpick 5MP dataset.

Image stitching: thanks to opencv gpu team, we now have image stitching in opencv! It is a separate application opencv_stitching that takes an arbitrary amount of images as an input and automatically generates a single image as an output. It is being tested now. A downsampled stitched image from 6 12MP images is attached (the pano series is quite challenging due to bright sun that changes lighting from image to image).

Other: Ilya and Andrey are on vacation this week, Alexander was on vacation May 16th.

Vadim

# (Documentation) Our tech writer finished grammar check of the reference manual. Now she is fixing the hyperlinks and will then check the user's guide and the tutorials.
User guide and the tutorials are included into the build process. Online version of the whole documentation (3 parts of the reference manual: OpenCV # x C++ API, OpenCV 1.x C API, OpenCV 1.x Python API, User guide, tutorials) is now automatically generated as well.

#  (Fixing bugs) 10 tickets have been closed + several build problems and failures in tests, discovered by our buildbot, have been eliminated.

plans.
# continue to clean the bug tracker.

Action Items

Gary

• Tell Vincent to put ORB in

Radu

Alexander

Vadim

Victor

• Get stitching to have test and doc

Kurt

Ethan

James

From last time

Gary

• Contact GSoC interns
• Get foils to Victor

Radu

Alexander

Vadim

Create a tutorial stub

• Contact GSoC interns
• Make sure the tech writer incorporates the opencv.willowgarage.com/wiki/documentation user changes.

Victor

• Get me a list of tutorials suggestions
• Contact GSoC interns

Kurt

Ethan

James

Agenda

• GSoC … vendor resolved?
  • Any intern issues?
• Release status, on schedule?
• Docs.
  • PDF doesn’t have detailed sections to jump to like the old one.
• Tutorials
• Future directions: More pose fitting
  • New circular calibration pattern.

Minutes

• GSoC vendor status
  • Sergey is now also an admin, hopefully will take care of it.
• Documentation
  • The table of contents for the pdf is now smaller. In return, there should be an index, but it’s not in yet.
  • Vadim will enable tables on every chapter header, imilar to the python documentation here.
• Alexander Mordvitsev (GSoC intern) is now working on further improvements to python bindings and on creating python
• Bindings for Java are being worked on by NVidia team.
  • 2 scripts for python:
    • 1st script parse OpenCV headers with macro annotation (input, output …)
    • 2nd script takes this list and creates C python wrappers from the output of 2nd script
    • Java just needs to write a 2nd script for Java.
    • Want to do this for Matlab eventually
      • Have to do this for swig as well, but here we have complete control
• Vincent
  • Commited alternative python bindings based on python boost, mat wrappers and some highgui
    • Hard to maintain, every function that changes in C++ then needs someone to update
    • No documentation on new bindings
      • Hopefully new docs will be there before the release
• So far, still on Schedule for June 20th release.
• Who to get for OpenCV debug hackathon.
  • James for python
• Tutorials in rst/sphinx
  • Can we embed youtube into rst such as http://countergram.com/youtube-in-rst
• TOD is used in Technical University of Munich and they found some problems
• DOT
  • Maria is putting in additional false positive
• Victor in US right after CVPR. Week of June 27

Vadim

# (Documentation)
- Our tech writer started the second round of the reference manual review.
- I'm writing a script that checks completeness and correctness of the reference manual (i.e. it should catch the cases when the number of parameters or their types have been changed). Right now the part that scans the reference manual and the part that extracts the API from headers (based on our parser for Python bindings) are ready and mostly work well. Comparing the two sets of API is in progress.

#  (Fixing bugs) 18 tickets have been closed.

#  (GSOC) Alexander Mordvitsev started work on Python bindings. Preliminary version of letter_recog.cpp that demonstrates the use of various classifiers from ml is ready. Apparently, the automatically generated and never debugged Python bindings for random trees works well!

Plans:
# continue to clean the bug tracker.
#  finish the script. Start updating the documentation.
#  start OpenCL+OpenCV GSOC project.

Victor

TOD:
- finished working on #5103 task. Added new mode -M 2 to the masker in tod_training package. Added ignoring SEG_FAULT signals using long_jump function. (51281 rev).
- fixed problem with zero distortion matrix in opencv_candidate (36818), fiducial (51513). [Alexander]

DOT:
- Made TBB and some other optimizations for DOT detection.
Table of detection times (ms) per template (on Intel Core i5 CPU 760@# 80GHz):

|  1 template      | 100 templates |
------------------------------------
DOT (with IPP, SSE)                            |         4        |        0.38   |
opencv DOT (last week - without optimizations) |        18.7      |        8.2    |
opencv DOT (this week - TBB and some other)    |        1# 4      |        # 4    |

Original DOT has been run without templates clustering, opencv DOT implementation has additional normalizations and checks. [*Maria*]

Buildbot:
- continue investigation problem with Python on MacOS. Tried to reinstall Python.
- removed access for writing on our local repository
- fixed problem with incorrect DNS on MacOS -> error on sync of repositories
- created backup our virtual machines and scripts
- installed Bullseye coverage system on Ubuntu virtual machine, investigated console tools for calculation of coverage [Alexander]

opencv:
- descriptor_extractor_matcher sample was modified to print a more accurate precision-recall curve. [*Maria*]

iOS port: the project has started. opencv # 2 was successfully compiled for iOS by Xiaochao, with the exception of tests, samples, highgui and haar_training app. Artem is working on iOS support in highgui.

Other: Ilya is on vacations this week.

Action Items

Gary

Radu

Vincent

• Revert python bindings to use the standard ones.

Alexander

Vadim

• Talk with Vincent about python bindings
• Talk with James about Python bugs, especially related to numpy

Victor

• Test code and documentation for DOT

Kurt

Ethan

James

From last time

Gary

• (./) Tell Vincent to put ORB in

Radu

Alexander

Vadim

Victor

• Get stitching to have test and doc

Kurt

Ethan

James

Agenda

• Release status
• GSoC status
• Post release preliminary plans

Minutes

• SIFT
  • Previously we used a previous version of Andrea’s / | VLFet until it moved to GPL. This was an accurate but slow implementation. And it has problems for Android build. Moved to:
  • Rob Hess’s code which is 10x faster, maybe just a bit worse in performance, but we can tune if needed.
• Bug hackathon.
  • 21+11+3+2 = 37 bugs fixed so far.
  • Putting an index table in the doc chapters is also on the list.
• Ana started the tutorials, but is using C not C++.
• Future work
  • Color constancy/color balance (see http://www.eecs.harvard.edu/~ayanc/color-constancy/source.html )
  • de-noising, noise estimation
  • stereo camera — normalize color between the two cameras
  • estimating illumination and its direction
  • Estimating objects from specularity and lighting direction

Vadim

# (Fixing bugs) The bug-fixing hackathon week started. 21 tickets have been closed by myself + Ilya, Maria and Alexander fixed quite a few.

#  (Documentation) The documentation checking script is in progress; it reports some false alarms which I'm  trying to eliminate now. Our tech writer continues the second pass through the documentation. Next week is supposed to be the documentation improvement hackathon.

#  (GSOC) Alexander continues with MLL Python samples; knearest sample has been written, 2 bugs in Python wrappers have been found (and already fixed). Salman is slowly starting with OpenCL project. The work will be done in a separate branch not to affect preparations for the release.

Plans:
# continue to clean the bug tracker.
#  finish the documentation check script. Prepare for the documentation hackathon.

Victor

OpenCV descriptors: Replaced Vedaldi's SIFT implementation by Rob Hess's SIFT implementation (#1029) due to license issues and speed. precision-recall curves for several OpenCV descriptors, including both SIFT implementations, on different images (graf, trees and wall) are attached to the message. The comparison of running times for Vedaldi and Hess implementations is below:
|  Vedaldi's SIFT |  Rob Hess's SIFT |
detection time per image 1000x700    |       1906 ms   |        1105 ms   |
description time per keypoint        |       0.83  ms  |        0.23 ms   |

The regression test for detectors and descriptors (detector_descriptor_evaluation.cpp) has been ported to gtest. [*Maria*]

OpenCV bug fixing: fixed compilation errors with Eigen3 (#805), fixed drawKeypoints (it has assumed upward y-axis direction in keypoint orientation drawing), resolved tickets  #727, #552, #573, #950. CMake scripts were modified for compiling OpenCV using MinGW in static mode. [Maria, Alexander]

More tickets closed:
obsolete: #920, #259,  #685,  #874, #410
fixed: #1063, #886, #797
worksforme: #1049, #118
invalid: #961 [Ilya]

Howard algorithm:
A simple modification of Howard algorithm finally allowed estimation on new_college precise enough for sba to close the loop. The algorithm uses 3d<->3d matching to estimate pose  on a maximal clique, and then refines it by recalculating the pose on inliers within the clique. Then the pose was further refined using sba, similar to pe3d. The refinement on inliers produces a huge difference on new_college sequence, causing loop closure. An image of an example trajectory is attached. The algorithm running time is comparable to pe3d, but can be significantly sped up. [Alexander]

Other: Anatoly and Victor participated in nVidia GPU Technology Summit in Israel. Anatoly gave a 40 minutes talk on opencv and gpu module, and Victor did a 15 min OpenCV overview in a parallel track. There was a lot of positive feedback on the talks and tons of questions on opencv. Anatoly cold hardly get lunch, talking to people for two hours after his talk.

Alexy Spizhevoy (GPU)

Accomplishments & statuses:

1) OpenCV & GPU
Implemented first version of GPU Canny (2-10x speedup).
Fixed several bugs in SURF_GPU.
Integrated stream support to NPP wrappers.
2) Image stitching
Reduced memory requirements in multi-band blender, it now creates border of 2^number_of_bands size approximately, instead of fitting source images to the final image size.
Fixed bug in initial focal length estimation, it's quite stable now.
Replaced brute-force matcher with FLANN matcher (in CPU mode).
Implemented two gain compensation methods (full image and block-based).
Added more flags into stitching app, added preview mode, help message.
3) Auto-calibration
No update.
Plans:
Continue with Canny on GPU.
Resume auto-calibration activity.

Action Items

Gary

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

James

From last time

Gary

Radu

Vincent

• Revert python bindings to use the standard ones.

Alexander

Vadim

• Talk with Vincent about python bindings
• Talk with James about Python bugs, especially related to numpy

Victor

• Test code and documentation for DOT

Kurt

Ethan

James

Agenda

• Python bindings
• Release schedule
• Future plans:
  • color constancy
  • lighting direction
  • de-noising, analysis

Minutes

• If Qt not turned on in cmake, no Qt will be linked. If you do not have Qt on your system, it won’t compile
  • gtk also
• Release still on schedule
• DOT inclusion is postponed since not optimized yet

Vadim

# (Fixing bugs) The bug-fixing hackathon is semi-over. Ilya, Alexander, Maria and me took part in it. Alexander and Maria continue fixing bugs. 37 bugs have been fixed, some more are in progress. The total number of open trac tickets has been decreased to 290. Many of them are enhancement requests, there are quite a few duplicates etc. but we will probably need another hackathon week to clean bug tracker a bit more before the release.

#  (Buildbot, tests, coverage)
Buildbot has been extended to measure function coverage (see http://pullrequest.opencv.org/buildbot/, report from the "coverage" builder) [Alexander]

Previously disabled highgui tests have been put back into the testing loop [Andrey]

#  (Documentation)
Documentation update hackathon has been started. 9 trac tickets related to the documentation have been closed [Ilya]
Documentation check script has been finished [*Vadim*]. Here is the summary of its work:

undocumented functions in core: 296 -- most false since many small functions that are for internal use only
undocumented functions in imgproc: 42
undocumented functions in calib3d: 58
undocumented functions in features2d: 388 -- most of these are false since the interface exists for all functions but is documented only once.
undocumented functions in video: 50
undocumented functions in objdetect: 86
undocumented functions in highgui: 25
undocumented functions in ml: 311

such big numbers are obviously false alarms, because the parser found a lot of tiny methods in the internal-use-yet-exposed classes (especially in core and ml). In features2d, for example, the script did not take into account that all the methods in various  detectors, descriptors and matchers need to be documented only once. Perhaps, some "whitelist" or "blacklist" file should be added to make the script ignore certain classes and methods.

#  (GSOC)
Alexander put as much as 4 Python samples to SVN (letter_recog.py (classification), gauss_mix.py (EM demo), coherence.py (advanced image processing), browse.py (advanced highgui use)).
Salman is working on OpenCL reimplementation of GpuMat.

(Miscellaneous)
Maria was on vacation for 3 days (moving to the new apartments)

Anatoly (GPU)

Accomplishments & statuses:
1)      GTS Israel 201# Talk at the conference about OpenCV on GPU.
·         There was many questions
·         Got positive feedback from NVidia guys
2)      OpenCV GPU and CUDA # 0. Switched our code to the new release.
·         Updated GPU Module to use new stuff in NPP# 0 and CUDA# 0 (all asynchronous functionality, cudaDeviceSynchonize, etc.)
·         Updated OpenCV GPU Wiki. More detailed installation guide (merged ours and Craig Duttweiler’s pages there), FAQ etc.
·         Removed Multi-GPU manager since it is not necessary with CUDA# 0. Multi-GPU samples were rewritten.
·         Tested GPU module under Windows and Linux with CUDA# 0 on GT275
o   Found new issue for Face Detection for under ubuntu64 for GT27#  Logs and dumps were sent to Anton.
3)      OpenCV GPU on MacOS. Compiled and tested it.
·         Prepared detailed description of MacOS compilation.
·         Fixed several compilation issues (that ffast_mat  workaround, samples compilation).
·         Found issues (filed tests and a crash).
o   May be this is CC1.1 problem, not just MacOS. So going to test OpenCV GPU for old cards also.
4)      PCL
·         Tried to look into the HLBVH code Joe send me. Can’t understand it quickly, need much more time. Learned what is SAH optimization and some papers about KD-tree on GPU.
·         Implemented Octree building on GPU, but found a bug when I started profiling it. Trying to find the bug.
Assuming that the bug does not influence on performance, I can give some performance numbers for Octree building.
PCL on CPU:  300K points  23ms, 3M points 216ms.
on GPU:          300K points  60ms, 3M points  118ms.
I am implementing Octree with my own structures, using some ideas from the HLBVH paper of 2010. I decided to do so because it is simpler, and I have understanding how to implement radiusSearch.
Sure the performance is not final.  I guess main problem for the small dataset is a kernel calls overhead. Approach from the paper of 2011 eliminates it (task queues). Just need to understand how to do it.
5)      Auto-calibration
·         Studying of libmv. This is a SfM and auto-calibration library, they have auto-calibration algorithm implemented.
·         Started implementation of general infrastructure for auto-calibration (finished bipartite graphs, convenient representation of reconstruction,…), some parts of code are based on libmv.
Just one notice. Auto-calibration complete implementation may take month or two. This problem includes some 3D reconstruction problem.
User will have to take a photos of close well-textured 3D object from different sides (landscapes are not suitable).
Joe, you said you had pinged guys which have pattern-less camera calibration implemented. Do you have any update?
6)      Other
·         Some help to Tegra team with OpenGL demo application. The Vlad has some skills and experience in OpenGL.
·         Minor optimization and improvements stitching application
Plans:
1)      OpenGL demo for Android
2)      More OpenCV GPU testing (different platforms, configurations and hardware, multi-GPU mode).  Also test GruphCuts from NPP# 0.
3)      Build binaries for all platforms OpenCV GPU and place them in opencv_extra and svn.
4)      Continue work on Auto calibration and PCL.

Gary

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

James

From last time

Gary

Radu

Vincent

Alexander

Vadim

• Talk with Vincent about python bindings
• Talk with James about Python bugs, especially related to numpy

Victor

• Test code and documentation for DOT

Kurt

Ethan

James

Agenda

• Release
• GSoC
• New Plans

Minutes

• Trip: Intelligent Vehicles
  • Talk, lots of questions
• Plan object recognition strategy when Victor is here.
• Victor here June 26 for a week.
  • Monday NVidia
• Release
  • Another week of bug solving
  • Seems still on time for June 20th
    • Android Java interface will not be ready. Release for Android will maybe be separate.
      • C and C++ level and camera API is OK
      • Java bindings are not complete. Samples working. No docs. Another month for Java bindings.
• Kinect calibrated with another camera

Vadim

# (Fixing bugs) Since the bug fixing hackathon did very well, the team decided to spend one more thing one the same fun stuff. Maria fixed 15 bugs, Ilya - 8 (+some docs), Alexander - 9, Vadim - 21.

#  (Buildbot, tests, coverage)
A lot of build problems on various platforms (partially coming from trac, partially discovered on our buildbot and workstations) have been resolved, especially mingw (32-bit and 64-bit) support. 5 test failures have been fixed too, 10 are remained to be fixed. [Alexander]

#  (Documentation)
Our technical writer continues the second round of the docs review. Most hyperlinks are now in-place. Doc checking script has been fixed to lower the number of false alarms. Some missing descriptions have been added. Now the numbers are (last week numbers => this week numbers):

undocumented functions in core: 296 => 213
undocumented functions in imgproc: 42 => 2
undocumented functions in calib3d: 58 => 8
undocumented functions in features2d: 388 => 367 (should be much lower, indeed)
undocumented functions in video: 50 => 0
undocumented functions in objdetect: 86 => 80
undocumented functions in highgui: 25 => 23
undocumented functions in ml: 311 => 86

#  (GSOC)
Alexander added calibration and video capturing samples. There is no any noticeable progress on OpenCL - perhaps, there will be after the release when all the infrastructure is set up and the student will be just adding functions, which should be rather monotonous task.

Anatoly Baksheev

1)      PCL Octree
·         Spend some time for finding the bug in Octree without success. It appears only with CUDA profiler and kernel that crashes is too simple to contain a bug (possible Thrust or CUDA profiler bug).
·         Investigated bottlenecks of the implementation. I think it’s kernel call overhead and a lot of small portions of work that can’t workload GPU entirely. I stopped development in this branch.
·         Understood code Joe send earlier with task queues. And found idea of global sync barrier from Duane Merrill’s code (the fastest radix sort author).
·         Started I implementation parallel octree building with tasks queues and only single kernel call. In order not to debug on GPU:
a)      Implemented   quasi-parallel   prototype on CPU, tested and fixed bugs.
b)      Implemented and tested radius search on CPU for my data structure
2)      GPU Module
·         Fixed gpu::cvtColor according to changes in SVN r5324 for CPU.
·         Fixed bothersome BFM warning (  nvcc cannot tell what pointer points to shared or global  ) with PTX insertion!
·         Tested GPU module (old cards, Multi-GPU, different systems). Some issues for Ubuntu3#  Guess will fix before release.
·         Tested GraphCuts from NPP# 0. Now it does not return TEXTURE_BIND_ERROR, but still fails with KERNEL_EXECUTION_ERROR. May be we don’t understand something. It would be nice if we can write to GraphCut’s author directly (Timo Stitch?).
·         OpenCV Release is planned next Monday. We decided to:
c)       Construct and publish binaries with GPU support for windows. Without CUDA Toolkit binaries (300Mb). So users have to install CUDA toolkit to run GPU.
d)      Tarballs for Ubuntu and for MacOS (sources).
e)      Mark in Release Notes, that GPU Module is beta or RC for MacOS due to lack of testing.
f)       Point in Release Notes, that Face Detection/NPP_staging tests fail and here might be issues. (Linux64/Cuda40/GT275, MacOS/CUDA40/9400M, for GTX470/Windows only integral images)
3)      Auto calibration
·         Finished working on auto-calibration infrastructure.
·         Implemented projective reconstruction for arbitrary number of views (RMS error is about 1 on Wedham College dataset: http://www.robots.ox.ac.uk/~vgg/data/data-mview.html)
·         Implemented dense protective bundle adjustment. (Sparse BA will be later).
·         Implemented linear auto-calibration (intrinsic camera matrices estimation) via DIAC estimation (dual image absolute quadric).  Currently it’s unstable and returning matrices vary much from run to run.
Issues:

a)      On some datasets projective reconstruction shows high RMS error probably due to noise and incorrect matches. Increasing matching confidence threshold helps, but it removes too much matches sometimes (with small number of matches projective reconstruction algorithm can’t work).
b)      Dense protective BA works slow (a few seconds per iteration on big datasets, but quite fast on small ones, depends on number of points in reconstruction) and it doesn’t decrease error.
4)      Other
·         Helped Tegra team with OpenGL demo application
·         Fixed incorrect behavior with different scales in opencv_stitching
Plans:

·         OpenCV GPU Release
·         Finish single grid Octree building. Investigate bottlenecks.
·         Try to increase linear auto-calibration stability. Implement metric BA. Implement protective reconstruction upgrading to metric one for further using with metric BA.
·         Canny (currently 2-10x speed-up), BFM radius Search optimization, equalizeHists (used for LBM Face Detection), etc.
Other:

I will be on vacations from 20 of June till 2 of July (Altai mountains tracking trip).

Action Items

Gary

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

James

From last time

Gary

Radu

Vincent

Alexander

Vadim

• Talk with Vincent about python bindings
• Talk with James about Python bugs, especially related to numpy

Victor

• Test code and documentation for DOT

Kurt

Ethan

James

Agenda

• Release
• Book
• Plans

Minutes

• Release
  • Release candidate is out now, see change log
  • Tutorials are in trunk, release is from branch rc# 3
  • Bug tracker
    • 205 bugs left down from >400.
      • 30-40 are “major” defects
      • rest are enhancements, patches or minor defects
  • New FLANN is slated to go into # # 1 since it’s too much code too close to release deadline
  • # # 1 will include new Android stuff (most notably, Java bindings to make OpenCV fully useful on Android).
    • Some optimization as well.
  • Stitching is in the new release.
  • Cheatsheet is updated.
  • Rest of the week:
    • Second pass of documentation, merging into one file
    • Docs … will soon have 5 manuals: C++, C, Python, Python old and Java
      • Want to autosync strip off headers, plug into rst (using documentation script)
        • Have tables of all the python functions
        • all the declarations will eventually be tabs
• Can we rsync opencv.itseez.com to opencv.willowgarage.com/documentation
• Make difference between cv::Mat and cv::Mat_ (just adds simple pixel access I(x,y) instead of I.at(x,y))
  • Other types without underscore Rect, Point, Size are integer. Rect_, Point_ Size_ can contain any type
• Turning on SSE4 does not auto turn on SSE #
• Future
  • Make OpenCV smaller
    • Link statically, only used function are in executable
    • Can there be a dynamically linkable core that is small? This relates also to the Khronos standard.
      • Need to establish this “core”
  • Book: Could easily overlay python with C++ functions
    • Samples are
  • Plans
    • Proposal to expand use of dynamic code generation with python — eliminate intermediate code in C++, would go to OpenCL directly which would run on GPU, CPU or to CPU on Sandybridge chips.
    • Code written in Python would then run 10x faster, say than hand crafted C code. This could also be done in other languages. Write algorithms in vector form (image algebra book, Ritter).
    • Calibration and other functionality. Intern implemented LePetit’s solvePnP
    • Bundle adjustment, model slignment
    • Push on object recognition. Line-Mod into Ecto, new quick quasi 3D features, transparent objects, model capture

Alexey Spizhevoy (GPU)

GPU module

Configured buildbot on machine with GPU (Ubuntu, 64bit).

Fixed building of GPU module under MacOS i38#

Refactored GPU module tests:

Print information about cards.

Run tests on all cards in system.

Split NVidia.multitest on several subtests.

Asked Timo Stich about issues in graphcut, he confirmed bug in NPP and said it works correctly, just returns error code.

OpenCV # 3rc preparations is almost finished, binaries will be available today here http://sourceforge.net/projects/opencvlibrary.

Plans:

Fix out of memory errors in performance sample on machines with weak GPU, add multi-gpu support.

Implement filters for big kernel size (ticket #1134).

Implement gpu::equalizeHist.

Add GPU based graphcut into opencv_stitching (later).

Stitching:
Added wiki page with usage and simple samples: http://opencv.willowgarage.com/wiki/opencv_stitching.
Auto-calibration

Updated inlier/outlier classification step (point is inlier if it’s possible to get some sub-reconstruction with low reprojection errors for all views where point is visible).

Added poor points removal after each iteration in projective reconstruction algorithm, it increased stability a lot.

Fixed some bugs in projective to metric upgrading step.

Built SBA library, installed PCL library (for viewing point cloud).

Issues:

After linear auto-calibration step and upgrading projective reconstruction to metric I get similar intrinsic matrices for all cameras after RQ decomposition (as expected), but sometimes result matrices differ in sign at some elements.

Stability of projective reconstruction is quite good, however there are bad datasets where it’s difficult to find good initial pair.

Often upgrading of projective reconstruction to metric gives poor or reverted visual results (however reprojection error is small) if we use linear auto-calibration only.

Plans:

Fix bug with incorrect signs in intrinsic parameters after upgrading to metric reconstruction.

Implement refinement of metric reconstruction (i.e. parametrized via intrinsic and extrinsic parameters) using SBA library. I think I’ll use some pieces of code from Bundler application.

Sample image 001.jpg from dataset and respective point cloud college.png (built by 5 images) after linear auto-calibration and upgrading projective reconstruction to metric one are attached.
PCL octree:
No noticeable update, development is paused because Anatoly Baksheev is on vacations from June 20 till Jule #

Action Items

Gary

• Talk with Nathan about rsync ’ing opencv.itseez.com to opencv.willowgarage.com/documentation

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

James

From last time

Gary

Radu

Vincent

Alexander

Vadim

• Talk with Vincent about python bindings
• Talk with James about Python bugs, especially related to numpy

Victor

• Test code and documentation for DOT

Kurt

Ethan

James

Agenda

• Release left overs
• Docs/book stuff
• Priorities
  • Object Rec
  • Khronos - we have a window of opportunity, we should use it -
  • Android slimmed down version subset of Khronos

Minutes

• Full # 3 release is out
• Switch opencv to Git soon
• Need to automatically copy new documentation to opencv.willowgarage.com/documentation site
• Khronos
  • Data structures are the first part GPU vs CPU
    • CPU: easy pixel access
    • GPU needs some kind of map for pointers for pixel access and then do unmap
• Question is whether we can make an image algebra that can cover existing and new functions.

Victor

OpenCV Release
OpenCV # 3rc has been released on June 21st, OpenCV # # 0 final has been released on July 4th (or July 3rd PST time). The announcements are:

# 3rc:  http://tech.groups.yahoo.com/group/OpenCV/message/81325
# # 0 final: http://tech.groups.yahoo.com/group/OpenCV/message/81509

The list of major changes is here:
http://opencv.willowgarage.com/wiki/OpenCV%20Change%20Logs

Highlights of this release are:
* Much, much better documentation, which is for now hosted at http://docs.opencv.org and updated nightly.
* Buildbot is now working very well, building and testing OpenCV in many different configurations.
* Better android support with camera module, demos etc. (with more stuff to come in # # 1)
* Improved GPU module and the new opencv_stitching application that can run on CPU only, but can utilize an NVidia GPU when it's available.
* New calibration pattern, new feature descriptors BRIEF and ORB.
* Improved and stabilized Python support, now with complete documentation with growing number of samples (the latter is the ongoing GSoC project)

Improved Documentation
3 reference manuals have merged together. Grammar, style of the look of the documentation has been significantly improved, thanks to the professional technical writer.

121 Fixed Bugs
The whole month the team has been working on fixing bugs and improving documentation. 121 tickets in OpenCV bug tracker have been closed during June, bringing the total number of open tickets to just over 200 (including minor bugs, enhancements requests etc.)

Vadim

# (OpenCV release: # 3 is finally done!) OpenCV # # 0 final has been released. The announcement is here: http://tech.groups.yahoo.com/group/OpenCV/message/81509. The main work has been done on the documentation - merging several reference manuals into one, and on the preparing windows package. Also, a few bugs in the # 3rc have been fixed. See below on each of the items.

#  (Documentation)
The 3 reference manuals, covering C, C++ and Python, have been merged together. Besides, the new-new Python interface is put there too. It's all available now at http://docs.opencv.org. Thanks to the shared description it will now be much easier to maintain the documentation. Besides, the Java interface description can be semi-automatically integrated there as well.

Our technical writer made some final touches to the documentation and reviewed the new material, added after merging the reference manuals.

#  (Windows package)
The huge problem on Windows, when using Microsoft compilers, is that different versions use different, incompatible runtime and applications built in debug have to link debug versions of DLLs, otherwise it will crash. I submitted a poll: http://tech.groups.yahoo.com/group/OpenCV/surveys?id=3051965, asking people what they want, and they appear to want pretty much everything. My multiplying all the needed variants: 4 compilers x 32-bit/64-bit x debug/release x static/shared x TBB on/off we get 64 variants of OpenCV! Plus we need at least 3 different Python wrappers. By excluding VS2005 and setting TBB=on in the case of DLLs we can bring this number down to 24 variants. Then the first package has been built, it was ~300mb archive. After a few days of tweaking make scripts, stripping debug information, making universal ffmpeg plugins, building Python wrappers statically (so they are absolutely self-contained), and by using a better compression the package has been brought down to ~100mb, which has been uploaded to SourceForge as opencv-# # 0-win-superpack.exe. It is not an installer, just a self-extracting archive.

#  (Bug fixing)
8 tickets have been closed this week.

#  (GSOC)
Alexander added 8 more Python samples. OpenCL project is stuck. Some first code is received, but it mostly a stub thing. Will make another attempt to motivate Salman to do something better.
*

_*Anatoly Baksheev*_
*
1)      opencv_stitching

·         Optimized building of warping maps using GPU in opencv_stitching. As result we have 2x speedup (
40sec -> 20sec) for the whole compositing pipeline in case of spherical warping 12 Mpx photos.

·         Optimized pyramid-related stuff with GPU and got ~ 2x speed-up for it. This does not have big
impact on total pipeline time.

o   May be will re-implement after, because we expected >10x (speed-up of Gaussian Blur on GPU), but got
only 2x. It seems that CPU pyramids are hardly optimized (hardcoded blur 5x5 that is mixed with
sampling, etc.), so we can get ideas for GPU here.


2)      Auto calibration & reconstruction

·         Implemented synthetic datasets generation framework (opaque spheres and cubes from points, rigid
camera matrices, "photos" with keypoints and correct matching).

·         Started implementation of tests for the whole auto-calibration pipeline. We want to cover the code
with a lot of tests in order to find most computationally unstable parts of it.


Issues:
·         Found that reconstruction pipeline can't handle scene with a cube and 15 cameras around the cube
in the (OXZ) plane (it can't continue after half of images).


3)      PCL

·         Implemented octree building with global barrier. Caught a lot of bugs, but not all. (Some blocks
of octree structures exactly match with tested CPU prototype, but total number of nodes differs).

·         Approximate work time for 870k points ~10ms (without memory allocations etc.). This may change in
future after bugs will fixes. PCL Octree is built for 75ms. Kirill Garanzha claims 8.1ms for
HLBVH in his paper.


Plans:
·         Find root cause of problems in reconstruction pipeline. Continue testing of auto-calibration.

·         Catch all bugs in octree implementation, refactor and clean code, trivial optimizations (
reduce memory allocations, etc.).

o   Implement radius search on GPU.

·         [low priority] Try to port original CPU pyrUp/ pyrDown implementation.

·         [low priority] Add support of different interpolation and border modes into gpu::remap,
as it's one of hotspots.

Action Items

Gary

• Talk with Nathan about rsync ’ing opencv.itseez.com to opencv.willowgarage.com/documentation

Radu

Vincent

Alexander

Vadim

Victor

• Get ftp site for copying documentation and access for us to automatically copy

Kurt

Ethan

James

From last time

Gary

Radu

Vincent

Alexander

Vadim

• Talk with Vincent about python bindings
• Talk with James about Python bugs, especially related to numpy

Victor

• Test code and documentation for DOT

Kurt

Ethan

James

Agenda

• Python bindings
• Khronos
• SFM/Bundle adjustment, SLAM
  • What is where
• GSOC reviews in
• Autodoc update

Minutes

• SLAM in kforge
  • mega-slam
  • multi-slam
  • Bundle adjustment
    • One for image stitching camera focus stays on the same 3D point so homography works for any 2 images (in image stitching app)
    • OpenCV has LM solver for camera calibration etc … bundle adjustment is just sparse LM (fast way of doing LM optimization)
      • GPU does dense LM taking advantage of the GPU
    • No bundle adjustment for monocular slam
    • For stereo SLAM, use the bundle adjustment in ROS, but has problems. (vslam stack).
      • vlsam: pose est has SFM, 3Dto3D R,T finder, SBA (sparse bundle adjustment), place recognizer … might make use of.
        • Also see http://openslam.org/ visual odometry and slam systems
• Khronos
  • Need to update the document
    • Vadim did experiments with OpenCL
      • Experimented with python bindings for OpenCL
      • Compose code in the form of a text string and then OpenCL compiler computes
        • For matrix multiply 1000×1000. OpenCL did 15x faster on GPU than OpenCV matrix multiply on CPU
        • On CPU it was 10% slower
• Android — need to get back to Rodrigo who is getting back early August (when I go on vacation …)
  • Train up high value classifiers
• Line-mod

Vadim

# (Bug Fixing) Despite that # # 0 is out, there are still quite a few (>200) open tickets in the bug tracker, plus the new bugs are reported on a regular basis. So, the work on cleaning the bug tracker continues. 14 tickets have been closed last week.

#  (Python bindings)
cv module (previous-generation bindings) has been embedded into cv#
Users of the cv module should replace:

import cv

in their code with

import cv# cv as cv

This is the only change to be made. After the change all our Python samples and tests run fine.

With the change we further improve OpenCV compile time and reduce the footprint of binary packages, especially on Windows, where cv.pyd and cv# pyd are built statically (i.e. they contain the whole OpenCV), so by getting rid of cv.pyd we reduce the package size noticeably.

#  (Optimization)
There have been several reports that OpenCV has a few performance regressions, comparing to 1.x and # 0. One of such report was about calcOpticalFlowPyrLK. The C++ version was several times slower than the C version. Last week the code has optimized using SSE2 and parallelized using TBB. As a result, the new implementation is >9x faster than the previous C++ implementation and ~2-3x faster than the C version. The C version has been dropped and replaced with a thin wrapper on top of C++ function.

#  (GSOC)
Alexander added 5 more Python samples, covering optical flow, motion templates, contours and face detection. OpenCL project is very slowly emerging. There is the first code drop in ocl branch, but it is far from being usable yet.

Victor

Model capture. Vslam has been enhanced to support bag files in tod format (synchronized topic with points, image_color, camera_info). Experiments with kinect bags show good camera trajectories but inconsistent point clouds. We are working to explain the latter [Alexander]

Several experiments with Meshlab to create a CAD model from a point cloud. Kinect was connected to Meshlab with an experimental plugin from the Meshlab trunk. Experimented with MeshLab to create a model of an object (it has several algorithms of surface reconstruction from a point cloud). Attached is the constructed model of the blue bowl when the object is captured from several close viewpoints (cloud.png, model# png) [Ilya]

Experimented with the rgbdslam ROS package developed by the University of Freiburg to get a point cloud of an object. It seems to work well for a large textued environment but accuracy is insufficient for creating a model of a small object. For example, we got several close point clouds of the object instead of single one. So the created model has a lot of artifacts (in the best case) -- see gicp.png. [Ilya]

Buildbot: changed script for uploading OpenCV HTML documentation to the FTP using ncftpput. Uploading time is decreased from 1.5 hours to 20 minutes. Now we uploading OpenCV trunk documentation to thehttp://docs.opencv.org/trunk, release version is still available on http://docs.opencv.org. Buildbot user interface has been moved to a separate workstation to minimize response time. [Andrey, Alexander]

OpenCV: Resolved the tickets #1206, #1179, #1162, #108. [*Maria*]

Other: the paper by Victor Eruhimov and Wim Meeussen "Outlet detection and pose estimation for robot continuous operation" was accepted for IROS 2011.
Maria was on vacations till July 8th.

Anatoly Baksheev

1)    OpenCV GPU
·         Fixed compile-time error in BFM test (it was revealed under ubuntu x64 in debug mode).
·         Fixed mean-shift filtering in response to Yahoo Group question, so that it support CC <=1.1 now
2)    PCL Octree
Accomplishments & statuses:
·         Fixed bugs in parallel octree building on GPU. So now I have working version. Speed-up is data and parameters dependent and varies in 10x-30x range.
·         Implemented radius search on CPU that uses Octree downloaded from GPU. This is preferable for several single queries. ~2x faster for best parameters.
·         Implemented but not debugged parallel batchRadiusSearchGPU. I guess I found efficient way to process leafs. Paused the activity in order to refactor and integrate code to PCL.
·         According to Radu’s request, refactored code a lot, dropped unused functionality, and prepared for PCL commit, integrated into PCL build system, write several simple tests, etc.
Interface is nice, but code inside is far from production quality. But I will commit today or tomorrow in order to get feedback from other PCL team.
Some performance numbers:
Data set:
871k points (uniformly distributed), volume size  1024f * 1024f *1024f.
10k queries for readiusSearch, with radius uniformly distributed in [0..1024f/15f].
Parameters:
For PCL Octree, resolution = 25f. Such resolution provides the best search performance. Build or search time rises drastically to seconds if change this.
For my OpenCV Octree contributed in 2009, max_points_per_leaf = 20, max_levels = 10.
For GPU octree, max_points_per_leaf = 10 (hardcoded), max_levels = 10 (doesn’t support other at all).
Hardware: GTX470 vs. Core i5-760 # 8Ghz
Build time: PCL – 232ms, OpenCV  351ms, GPU – 19ms(build) + 11ms (download to CPU for radius search there).
Numbers don’t include point cloud construction time and upload to GPU.
Radius search:  PCL  650ms, OpenCV – 383ms,   OctreeGPU.radiusSearchCPU (CPU search using downloaded octree)  –  382ms.
Please note that PCL returns vector of distances, but my implementations don’t.
Plans of PCL:
·         Finish batchRadiusSearchGPU.
·         Optimization of Octree building and batchRadiusSearch. (Have a lot of ideas).
·         Bring code to production quality.
·         Probably start knnSearch?
·         May be try parallel brute force GPU radius search to understand how and where octree helps (fast to implement).
Also I have ideas to investigate possibilities of fusion octree building with SRTS radix sort into one algorithm. I guess there might be very good finds and performance like 4ms to build 871k points. But I think I don’t have time for it.
3)    Auto calibration
Accomplishments & statuses:
·         Tried gold standard triangulation described in Multiple View Geometry (Zisserman) instead of one described in Pollefeys’s tutorial. It doesn’t affect results very much.
·         Fixed bugs in merging based and incremental projective reconstruction pipelines. Both now work correctly on synthetic datasets without noise.
Just some numbers for merging based reconstruction pipeline. We use synthetic datasets, i.e. out data is without measurement noise.
Linear auto-calibration for initial K = (Description: image.png ) = (Description: image.png ) returns K_result = (Description: image.png ), while ground truth = (Description: image.png ).
Adding small Gaussian noise (mean=0, stddev=0.1) changes results in the following way:
a)      for cube dataset calibration matrices are different for different cameras (e.g. mean of focal length=434, stddev=267),
b)     for sphere dataset results are much better (focal length mean=743, stddev < 1, i.e. returned results are correct).
Increasing noise (stddev = 1) makes results for cube dataset even worse, results for sphere datasets are still good (focal length mean = 707, stddev = 18).
For incremental projective reconstruction pipelines final reprojection error is higher and auto-calibration works worse.
Issues:
·         Spent two days trying to make work simple auto-calibration described in “A Simple Technique for Self-Calibration” paper. No success. Perhaps Lev.-Marq. method isn’t good enough for such purposes (quasi-Newton method is used in the paper).
·         Implemented auto-calibration technique is still unstable on noised/real data. Many researches also states that and trying to do something with it. There are few candidates to be implemented in papers.
Plans for Auto Calibration:
·         Try non-linear refinement of auto-calibration results proposed in “Robust Metric Reconstruction from Challenging Video Sequences” paper. Authors say it stabilizes auto-calibration process (they use the same linear auto-calibration as I).
·         Try to improve merging based reconstruction pipeline by implementing robust merging scheme described in “Accurate Camera Calibration for Off-line, Video-Based Augmented Reality” (they use MLESAC with non-linear refinement). For now I have linear merging only.
·         Try to apply any open source implementation of quasi-Newton method for another auto-calibration technique (from “A Simple Technique for Self-Calibration”).

Action Items

Gary

• (./) Talk with Nathan about rsync ’ing opencv.itseez.com to opencv.willowgarage.com/documentation
• Send out reminder for GSOC reviews
  • Tell Radu
• Help Maria get on kforge for the line-mod code

Radu

Vincent

Alexander

Vadim

Victor

• (./) Get ftp site for copying documentation and access for us to automatically copy
• Write Kurt for access to kforge code

Kurt

Ethan

James

From last time

Gary

• (./) Talk with Nathan about rsync ’ing opencv.itseez.com to opencv.willowgarage.com/documentation

Radu

Vincent

Alexander

Vadim

Victor

• (./) Get ftp site for copying documentation and access for us to automatically copy

Kurt

Ethan

James

Agenda

• OpenNI “sucks” when rectified. Patrick solved this for ROS
• OpenCV V2 started … convert code to C++
• Khronos?
• SFM
  • Talk to Vincent about SFM, stitching and blending code
  • GSOC intern working on libmv
• Other themes of the next release?
  • Object recognition
    • specialized cascades for humans
    • human recognition via MS method with the kinect
    • Line-mod?
  • Good module working with kinect
  • Khronos
    • API

Minutes

• Linemod
  • Does quite well, but multiple limitations
  • No save, no multiple obj in same frame, occlusion problems, not modular (parts based), needs help in auto-generating scale and rotation, scaling the learning
  • Sparse template, how to scale?
  • The code is really just one big main function, the functions are not modular, they are mixed gradient+
• File bug report on openni drivers when rectified
• libmv as basis of SFM in openCV
• Releases
  • Next patch release has Java binings and some Android demos and bug fixes
    • Android or Desktop or both this release?
  • # 4 Plans
    • line mod
    • tod
    • sfm
    • Cendric had reimplemented MS pose detector for kinect
• MS computer vision summer of code
  **

Vadim

# (FLANN) FLANN 1.6 has been re-integrated to trunk. "miniflann" template-less and Python-, Java-friendly interface has been introduced. It is 99% compatible with the normal FLANN interface (in Index::radiusSearch() call 1 extra parameter needs to be passed to fix the remaining 1% of issues). Now FLANN can be accessed from Python.

#  (Bug fixing; Python bindings)
11 track tickets have been closed. Several modifications have been made to Python bindings, OpenCV headers, and the bindings generator to make it more consistent with the upcoming Java bindings.

#  (SVN => Mercurial)
Since OpenCV development becomes more and more distributed; 3 teams at Itseez are working on it, plus there are GSoC projects and some new functionality is being developed at WG, it makes sense to transfer OpenCV repository to a distributed code management system, such as GIT or Mercurial. Andrey and Alex made seminars about the tools, from which Mercurial appears like a good first candidate to consider. Since code.ros.org does not support Mercurial, we've created experimental OpenCV project a kforge.ros.org (with the help from Eitan), where I uploaded the converted Mercurial repository. If we manage to migrate our bug tracker there (or, as an alternative, just fix it down to zero size :) ), we can migrate to kforge.ros.org and enjoy distributed development process.

#  (GSOC)
Alexander (Python samples) has passed mid-term evaluation and continues working on the samples (and even doing some fixes in Python bindings). Salman (OpenCV+OpenCL) failed.

#  (OpenCV functionality and API for the standard)
Just started working on the document. First version should be expected by the beginning of the next week.

Victor

Model capture: vslam stack has been adapted to take data from kinect (image+point cloud as opposed to 2 images from a stereo camera). The code works on bag files captured by kinect, but there are a few issues: 1) kinect depth map can be obtained from 0.5m, and the color camera is low quality so often we don't get enough texture to track the object through the whole bag file. We have plugged into vslam a pose estimator that uses a circle grid template finder to initialize camera pose. However the template is not seen on all frames. 2) Even when we calculate camera trajectory based with the template finder, we get few points in the point cloud since we track only textured features. Here we will have to move to an ICP-based dense bundle adjustment (talked to Nikola about this). [Alexander]

Experimented with 3D scene reconstruction using Kinect RGB Demo showed similar results to the rgbdslam package: it can create a good model of a large textured environment but the accuracy is insufficient for object modelling. Experimented with point clouds registration using Meshlab and Scanalyze (developed in the Stanford by Kari Pulli and Matt Ginzto). A dataset with 8 Kinect images of the 'objects on a table' scene was created. The step between frames is approximately 45 degrees so they span all 360 degrees of viewpoints. Point clouds have been registered using 3D coordinates only. Automatic registration from Scanalyze has not succeed so they were roughly aligned manually (later it will be done by VSLAM) and then are registered globally by Meshlab. The result is in registration.png attached. The model of the whole scene has been created using Meshlab: smooth.png, triangulation.png. [Ilya]

Untextured object detection. Experimented with LineMod code to study the algorithm capabilities for textureless object detection problem.
1) Trained detector online and detected some objects using original code (it gets rgb image and depth map from Kinect).
- The detection is robust given 3 objects (see orig_linemod.png).
- The algorithm for segmenting the foreground object does not work on our bag files -- the points within a rectangle around the object are required to be textureless and lie in the same plane.
- It is not designed to detect several objects of the same class, selecting a single hypothesis with the maximum confidence.
- An ability to save a trained detector is not implemented in original code (it's very inconvenient to retrain a detector every time).
2) We have modified LineMode code to run it on plates_cups_and_bowls base (this base has been used to test DOT):
- Tried to replace Kinect rgb and depth images by the corresponding images and point clouds from the training base. The object mask was obtained from the tod masker (we set rgb pixels masked as "background" to the constant color).
- Implemented the detection hypothesis filter that allows several detections per training frame.
- An example detection is in the attachment (linemod-detection.jpg). The results are much better than what we had with DOT. In the most of cases an object existing in training base was detected correctly or as very similar object. [*Maria*]

Anatoly Baksheev (GPU)

1)      PCL Octree

·         Implemented batchRadiusSearch on GPU. Did several optimization iterations. Speed-up is about ~10-20x. Found and fixed ~5 difficult bugs.

·         Fixed compilation issues under Ubuntu.

·         Updated the code to product quality, refactored tests, and committed changes.


2)      Auto calibration



Implemented:

·         Added non-linear refinement into auto-calibration process.

·         Replaced RANSAC with MSAC in reconstructions merging code.

·         Implemented robust version of auto-calibration. It tries different subsets of projective cameras.

·         Added a priory information handling into SBA process (zero skew, principal point at image center). SBA library doesn't allow to change cost function in easy way, so had to add extra terms via "virtual" points and its projection code.



Research & studying
·         Found the explanation why projective reconstruction (and as result auto-calibration results) can be poor on some Oxford datasets in the paper "The Problem of Degeneracy in Structure and Motion Recovery from Uncalibrated Image Sequences" (only or almost only co-planar points is degeneracy case for fundamental matrix estimation). Guess it explains poor results on the synthetic cube dataset, because the first image in the sequence is cube face.

·         Tweaked parameters a lot. Now both merging and incremental based reconstruction pipelines return quite good projective reconstructions, which can be used for auto-calibration on non-degenerate datasets. On non-degenerate cases it never fails, but auto-calibration quality depends very much on initial intrinsic parameters.



Issues:
·         Wedham College is the best dataset amongst all Oxford datasets. Other ones requires hard parameters tweaking or contain degeneracies.  It will be hard for user to create images sequence without critical motion subsequences, degeneracies, with optimal base-line, and so on. Think about using video for camera calibration, because video contains redundant data and good sequences for reconstruction can be extracted from it automatically (e.g. one method is described here "Robust key frame extraction for 3D reconstruction from video streams").


Plans:
·         Still have some optimization ideas to check for both Octree building and batchRadiusSearch (low priority).

·         Add possibility to pass array of pcl::PointXYZ to OctreeGPU. Or may be other point types from PCL CPU part. I didn't do it before, because of unclear OctreeGPU user's needs.

·         Think if it is possible to implement knnSearch on GPU efficiently. Think what else I can implement on GPU from PCL::Octree CPU functionality. And talk with Radu about next PCL task.



·         Find/create video datasets for autocalibration.

·         Implement extraction of good image sequences from video.


Other:
·         Most of OpenCV team including me will be on vacations for Computer Vision Summer School (by MS Research) in Moscow for a week since 27 of July.

Action Items

Gary

• (./) Talk with Nathan about rsync ’ing opencv.itseez.com to opencv.willowgarage.com/documentation
• Get Ethan to file a bug report on openni drivers
• Have Vincent talk sfm with Victor et al

Radu

Vincent

Alexander

Vadim

Victor

• Write Stephan and CC the rest about refactoring the linemod code

Kurt

Ethan

James

From last time

Gary

• (./) Talk with Nathan about rsync ’ing opencv.itseez.com to opencv.willowgarage.com/documentation
• (./) Send out reminder for GSOC reviews
  • (./) Tell Radu
• (./) Help Maria get on kforge for the line-mod code

Radu

Vincent

Alexander

Vadim

Victor

• (./) Get ftp site for copying documentation and access for us to automatically copy
• (./) Write Kurt for access to kforge code

Kurt

Ethan

James

Agenda

• Lots of documentation does not index well on opencv.itseez.com
  • example search on logpolar, or getaffinetransform click on the link, it only goes to the top of the section, not to the logpolar description
• What functions are not in C++? Logpolar is one.
• It would be nice to include in the documentation, what headers go with what
• HoG Filter — in GPU, test is in C++, right?. Which exact algorithm?
• Same for Latent SVM … might want to use this.
• Plans
  • Line-mod, too many people?
  • Itseez move to sparse features??
  • People detection
  • vSLAM
  • Model capture

Minutes

• Documentation
  • Some functions (Camel case), the search results go to the top of the page, not to the function
    • This is a case sensitive Java script problem
    • Don’t know how to fix …
• Good to get linemod in, Patrick working on it
  • How to scale is the research issue
  • Making a tracker out of it is also a bit of research for how to use a que of templates
• Kinect
• Rectified images glitch
• Want a way to register a better camera with the kinect
• Nice to get projector-camera(kinect) registration
• Converted 2, 4, 5, 6, half 7, 8, 9(1), 10 (easy)
  • 3 (Adrian), 11, 12, 13
• HOG
  • C++ and GPU, but no documentation in C++
  • Dalal and Triggs detection part. Our descriptors == standard ones so we can use their SVM vectors (linear SVM)
• Felzenschwalb
  • We just implement the detection part.
  • We don’t have training code and the format that we use is different and no conversion tools
  • Needs images and bounding boxes and it produces several SVM (each part: size rectangle, weights of the part)
  • Template with 2 views with 5 parts, would need 12 rectangles and 12 vectors
    • OpenCV wants sizes of main rectangles and parts, and the decision vectors for main templates and parts
  • University work
    • Cascade version of Felszenschwalb is nearly done
      *

Vadim

# (OpenCV functionality and API for the standard) The document has been prepared and sent last week. It is also attached to the report. It lists the most important OpenCV functions and algorithms. Each algorithm is ranked in terms of computational complexity and potential scalability.

#  (OpenCV book v2)
Converted samples from ch5, ch6 and ch8, as well as parts from ch7 and ch10.

#  (SVN => Mercurial)
The migration is on hold, since Nathan should make a dump of our bug tracker.

#  (Bug fixing)
6 bugs have been fixed + a formatting of opencv2refman.pdf has been fixed and improved.

#  (OpenCL)
Continued exploring OpenCL. The abstraction layer for OpenCV is being written that will cache compiled kernels between function executions and probably even between program executions.

Victor

Linemod: Implemented saving/loading of linemode templates to .xml. Probably, this implementation and file format will be changed after linemode existing interfaces refactoring or by other reasons, but we need persistence functions to speed up our experiments with linemode. We don't store the algorithm parameters, because we works with the default now.

Implemented another hypotheses generator that returns multiple hypotheses per location. The previous algorithm returned a single hypothesis for each location and often mixed up similar objects. Now linemod returns all classes that have high score for a specific location (see hyp_several.jpg). [*Maria*]

Model capture: We were able to get a trajectory of a kinect and the corresponding point clouds from VSLAM. Two patterns with grids of circles are used for pose estimation that is plugged into VSLAM through standard posest::PoseEstimation interface (instead of pe3d). VSLAM generates a single bundle-adjusted point cloud but it is not enough for us since the cloud contains only textured points found by a feature detector. A planned solution is to have an ICP-based bundle adjuster but meanwhile we use Meshlab for this: the point clouds from each frame are transformed into the same reference frame according to the position found by vslam, then they are exported to meshlab that has its own global registration algorithm. This is how we get a dense point cloud and then create a triangulated model. Several details:
- Experimented with ORB detector/descriptor within vslam framework. Got similar inlier count in comparison with FAST/Calonder, FAST/SURF.
- Added filtering of the table and background
- Added working with two circle grid templates for full object scanning.
- Picked up SimpleBlob detector parameters, after that the detection rate of the circle grids is ~99% [Alexander, Ilya]

Issues with Kinect:
Kinect noise was analyzed by performing a following experiment: a flat empty wall in the office was captured by Kinect several hundred times after sunset and standard deviations of depths in each pixel were computed. It was found a Kinect has a specific pattern of noise consisting of several vertical bands (std.png, black is zero deviation and white is 10 mm). Also it was found that the first measurements are the most noisy: the noise can be reduced by waiting 30s before capturing Kinect data (stdWithWait.png). Then the accuracy of Kinect relative to distance was estimated by taking mean of these standard deviations (accuracy.png).

So practical recommendations for object modelling are:
# Put object at 0.5 - 1 meters away from a Kinect.
#  Capture one frame and wait 30s before capturing following frames.
Unfortunately, the pattern of Kinect noise looks unpredictable so we can't use it to filter out noisy measurements. [Ilya]

Experiments with kinect superresolution: we made an an attempt to reconstruct higher-frequency 3D details from a series of kinect frames. Farneback is used for dense optical flow, the algorithm is similar to S. Schuon, C. Theobalt, J. Davis, S. Thrun, LidarBoost: Depth Superresolution for ToF 3D Shape Scanning, CVPR 2009. See animated gif sr.gif for results (800 images were acquired from almost the same viewpoint and images were resized with # 71 factor). This algorithm can be applied to get a depth map with high-resolution but currently it has too many artifacts on textureless areas.

Anatoly

1)      OpenCV GPU
·         Implemented cv::gpu::pow (for images). Investigated differences with CPU implementation (GPU is smarter and better) and corrected tests.
2)      PCL Octree
·         Redesigned internal structure of OctreeGPU, so that now it uses aligned pcl::PointXYZ.  Before conversion to pcl::gpu::PointXYZ was required. In future I am going to follow this design and suggest Radu doing the same.
·         Implemented version of batchRadiusSearchGPU that uses separate radius for each query in passed batch.
·         Implemented approxNearestSearchHost and approxNearestSearchBatchGPU.  GPU batch version ~50x faster than Octree on CPU but returns slightly worse results:
o    for 1/3 of queries results are closer than from PCL,
o    for 2/3 farther . Let  Dpcl = distance(query_point, results_pcl_cpu), Dgpu  = distance(query_point, result_pcl_gpu). Average(Dpcl Dgpu) = 13, when Octree voxel size = 102# f x 102# f x 102# f;
·         Added tests and performance tests for new functionality.
·         Started brainstorming how to implement exact knnSearchBatchGPU.
3)      Auto calibration
·         Implemented robust key frame selection scheme according to “Robust key frame extraction for 3D reconstruction from video streams” paper. On dataset we found (Medusa video sequence) it leaves too few frames to start reconstruction. More parameters tweaking is required.
·         Implemented naive 2 views auto-calibration (closed-form solution using fundamental matrix only, according to “The Geometry of Multiple Images”). Going to compare it with our auto-calibration framework.
·         Implemented optical flow based matcher, as it can be useful in video sequences auto-calibration.
·         Measured auto calibration accuracy data on some real/synthetic datasets. Conclusions:
o   Auto-calibration consists of two steps: finding projective reconstruction and upgrading it to metric one. The second step works ideally on good projective reconstruction from first step.
o   Auto-calibration works very good on synthetic datasets, if the datasets aren’t degenerate and doesn’t contain critical motion subsequences. Focal length and principal point coord. relative error is 1-3% (some tables attached).
§  We spend some time investigating why our previous synthetic sphere dataset had such degenerate cases.
o   There is only one good real dataset Merton College III (screenshot below) among four tested, where auto-calibration produces stable results (tables attached). Other datasets are not so good, perhaps due to degeneracies and/or small baseline.

o   Issues are, most likely, in projective reconstruction step (first step). So going to debug it and investigate other approaches of finding projective reconstruction.
Plans:

·         Debug auto-calibration on synthetic and real datasets (in order to find not bugs, but different degenerate cases, sacrifices of accuracy, etc.)
·         Brainstorm how to implement Octree knnSearchBatchGPU
·         Discuss with Radu PCL GPU interface.
·         (low priority) Implement fast pyrUp, pyrDown on GPU (current trivial GPU implementation isn’t faster than CPU one, because the CPU version is super-optimized). Required for opencv_stitching on GPU.
Other:

·         OpenCV team will participate for Microsoft Computer Vision Summer School in Moscow for a week since 27 of July.

Action Items

Gary

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

James

From last time

Gary

• (./) Talk with Nathan about rsync ’ing opencv.itseez.com to opencv.willowgarage.com/documentation
• (./) Send out reminder for GSOC reviews
  • (./) Tell Radu
• (./) Help Maria get on kforge for the line-mod code

Radu

Vincent

Alexander

Vadim

Victor

• (./) Get ftp site for copying documentation and access for us to automatically copy
• (./) Write Kurt for access to kforge code

Kurt

Ethan

James

Agenda

• Most people still on August vacation, just some updates

Minutes

• Version 2 of Learning OpenCV now has
  • Complete code conversion (for V1 functions)
  • 2 chapters written of ~13
• # # 1 release to support Java/Android
• ~100 bugs fixed since release
  • ~30 bugs fixed in August
• New tutorials (up to date is on)
  • http://docs.opencv.org/trunk/

Vadim

# (OpenCV # # 1; bug fixes)
All the past 3 weeks have been spent in preparations to # # 1, which should be out in a day or two.
26 tickets in the bug tracker have been closed + 2 crashes in our tests have been fixed.

#  (Planar subdivisions in C++)
Planar subdivision have been ported to C++. The previous C interface has been extended with some useful functions, like "get the list of triangles", "get voronoi cells". Delaunay sample was converted to C++.

#  (Contributions)
Alexandre Benoit contributed the code with retina model (http://www.lis.inpg.fr/pages_perso/benoit/resultats_demos.html.en). It was put to contrib module.

Marian Zajko contributed support for Ximea cameras (www.ximea.com).
The code has been put to highgui module.

#  (GSOC: Python Samples)
3 new samples added: HOG people detector, flann and camshift. Started the work on creating shell (meta-demo, that will contain a list of samples with their descriptions).

Victor

Model capture: experiments with vslam and various combinations of detectors/descriptors still show unstable tracking that results in a point cloud with multiple copies of the object or parts of the object (such as cup handles). Experiments with DWO-SLAM also show poor tracking quality. We have integrated tracking with circles grid templates into both slam codes to improve pose estimation, and this improves the quality of the final point cloud but still is not enough for model capture.

Untextured object detection: Captured a larger dataset of 27 untextured objects in TOD format, including metal knifes and forks. Experiments with linemod on this dataset are on the way.

Transparent object detection: captured a dataset of 12 transparent objects. Stereo data was captured using a painted copy of each object (see attachment). Experiments with 2D edge model fitting on this dataset are on the way.

OpenCV: Improved CMake script finding OpenNI (#1190, #1252), fixed memory leak in SIFT wrapper (#1288), implemented read/write methods for classes SimpleBlobDetector and DenseFeatureDetector and added an ability to create their objects using FeatureDetector::create method (#1290).

Other: Victor is on vacations till August 29.

Anatoly Baksheev

Accomplishments & statuses:

1)      OpenCV GPU
·         Minor refactoring & fixes(gpu test failure on empty test data, moved GpuMat class to separate header file, etc.)
2)      OpenCV stitching
·         Minor stitching optimization & refactoring’s (improved buffer reuse, updated command line interface to allow user to turn off bundle adjustment, etc)
·         Implemented optimized version of gpu::BruteForceMatcher::knnMatch for k == 2 (~2x speedup). This improves stitching performance for low- and middle- resolution images.
·         Added saving of matching graph in DOT format.
·         Added CameraInfo class (for future handling initial values of rotation matrix).
3)      PCL Octree GPU
·         Added support of pre-Fermi (emulation of some Fermi-intrinsic, modified kernel launch configuration, etc.). This was done according to Andreas Mutzel’s request.
·         Implemented and debugged knnSearchBatch for OctreeGPU. K is fixed to 3#  Current implementation works significantly slower (>10x) than CPU version. Profiling and trying to optimize it.
4)      Auto calibration
·         No update. Last status was: algorithms works on synthetic datasets, but don’t on real data.
·         Plans to find-read-try other approaches of constructing projective reconstruction. May be prepare summary of the project when Victor returns from vacations.
Plans:

·         Continue Octree::knnSearchBatchGPU optimization.
·         Read the following paper: “From Pictures to 3D: Global Optimization for Scene Reconstruction” by Manmohan Krishna Chandraker about SaM and global autocalibration, maybe try to implement some ideas.
·         Add handling of rotation matrix initial values into opencv_stitching, because they are available on mobile devices from sensors (we talked about it with Peter Karasev).
·         Re-Implement fast pyrUp, pyrDown on GPU. Required for porting some left components of opencv_stitching on GPU.
·         Optimize memory usage in gpu::BruteForceMatcher::knnMatch when k == #

Action Items

Gary

• (./) Talk with Nathan about rsync ’ing opencv.itseez.com to opencv.willowgarage.com/documentation
• Get Ethan to file a bug report on openni drivers
• Have Vincent talk sfm with Victor et al

Radu

Vincent

Alexander

Vadim

Victor

• Write Stephan and CC the rest about refactoring the linemod code

Kurt

Ethan

James

From last time

Gary

• (./) Talk with Nathan about rsync ’ing opencv.itseez.com to opencv.willowgarage.com/documentation
• (./) Send out reminder for GSOC reviews
  • (./) Tell Radu
• (./) Help Maria get on kforge for the line-mod code

Radu

Vincent

Alexander

Vadim

Victor

• (./) Get ftp site for copying documentation and access for us to automatically copy
• (./) Write Kurt for access to kforge code

Kurt

Ethan

James

Agenda

• Release
• Khronos
• Xilinx
• GSoC

Minutes

• # # 1
  • Java interface for Android
    • Desktop Java possible with additional work
  • 100 bug fixes
  • New version of FLANN
  • Improved python bindings and more samples
• Android team will soon have performance monitors
  • This will help us optimize performance on Android
• 3D reconstruction
  • Andew Davis approach: form real
  • Replicate Kinect Fusion
    • 3-4 papers with promising results
    • CTG — high quality dense reconstruction
    • For sure we need bundle adjustment
      • Differences between algorithms Jacobians analytic or numeric
      • Monocular vs Stereo settings
      • Talk with Kurt
• Intern with Xilinx doing vSLAM? Also camera and projector systems
• Alexxander B. To evaluate his student while he’s on vacation … Demos with this
• iOS … commits … don’t know. Aug 18th commit was latest. Need reports
• Tutorials went pretty well — both students passed of course.

Vadim

# (OpenCV # # 1)
OpenCV # # 1 is out. 7 more tickets have been closed.

#  (GSOC: Python Samples)
Alexander Mordvitsev created preliminary version (to be polished this week) of the demo shell that lists the samples. User can see description of each sample and run it.

#  (Sparse Bundle Adjustment)
Started yet another attempt to implement sparse bundle adjustment in OpenCV. The goal is to replicate functionality of sba package (http://www.ics.forth.gr/~lourakis/sba/). The algorithm core that inverts the sparse (J'*J+mu*I) matrix has been written and debugged.

Victor

Model capture: results with KLT+ICP tracker have been improved after parameter tuning. However the model is still far from perfect (see attached examples of captured models camera_model.jpg and cup_model.jpg). Tracking fails when texture is insufficient, noisy cloud, insufficient accuracy of trajectory obtained from template pose estimation.

Transparent object detection: evaluated the edge pose estimation algorithm on transparent objects. If the initial pose is not too far away from the ground truth and the clutter is limited, the pose is successfully reconstructed. Attached are images of a point cloud of a transparent glass together with symmetry axis that shows pose: originalAxis.png -- initial pose, finalAxis.png -- final pose.

Untextured object detection: evaluated linemod on white_objects base (this is a base of 12 painted transparent objects).
- Test data: We have 2 test bases for such objects: transparent and painted. We used painted test base. Implemented the masker for the test base to evaluate the detection rate. The masker registers point clouds of objects from all test images (of a given bag file) in abstract reference system, filters out the table and outliers and clusters the filtered point cloud on several objects, then projects each cluster to each test camera pose to get the masks. This masker works correctly if objects are sufficiently large and not too close to each other. With some manual postprocessing now we have a test set with correct object masks.
-Test system: Implemented an evaluation system for object recognition on given format of test base. Evaluation is performed for linemod with 2 types of hypotheses filers: one hypothesis per location and several hypothesis per location. The results for one hypothesis per location are below.

Obviously, linemod is a good choice for classifying a limited number of untextured objects. Also, find attached examples of objects recognized by linemod (1-# jpg).

Legend:
count - Count of given class objects in the test base (in all test images).
confusion matrix - Row index corresponds actual object; column index - detected object.
TPR - True positive rate = correct detected objects / count
FDR - False discovery rate = incorrect detections / all detections
O_FDR - False discovery rate on objects of incorrect class = incorrect class's detected objects / all detections
BG_FDR - False discovery rate on background = background detection as object / all detections

count     conf. matrix                       TPR          O_FDR      BG_FDR         FDR
background   nan [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
sour_creem   70  [35, 2, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]  0.5          0.054054055,  0,      0.054054055
small_bowl   86  [0, 64, 0, 0, 0, 1, 0, 0, 0, 21, 1, 0]  0.74418604   0.030303031,  0,      0.030303031
salad_bowl   33  [0, 0, 30, 0, 0, 0, 0, 0, 0, 2, 1, 0]   0.90909094,  0.23076923,   0,      0.23076923
form         44  [0, 0, 9, 37, 0, 0, 0, 0, 0, 0, 0, 0]   0.84090906,  0,            0,      0
cup          33  [0, 0, 0, 0, 17, 0, 0, 16, 0, 0, 0, 0]  0.5151515,   0,            0,      0
wineglass    77  [0, 0, 0, 0, 0, 62, 0, 8, 0, 0, 1, 0]   0.8051948,   0.015873017,  0,      0.015873017
bottle       63  [0, 0, 0, 0, 0, 0, 59, 0, 2, 0, 0, 0]   0.93650794,  0.048387095,  0,      0.048387095
glass        70  [0, 0, 0, 0, 0, 0, 0, 70, 0, 0, 0, 0]   1,           0.48148149,   0,      0.48148149
bucket       35  [2, 0, 0, 0, 0, 0, 3, 0, 33, 0, 0, 0]   0.94285715,  0.083333336,  0,      0.083333336
sushi_bowl   44  [0, 0, 0, 0, 0, 0, 0, 0, 0, 44, 0, 0]   1,           0.36231884,   0,      0.36231884
tiny_bowl    77  [0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 70, 0]   0.90909094,  0.04109589,   0,      0.04109589
bank         63  [0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 54]   0.85714287,  0,            0,      0

Anatoly Baksheev

Accomplishments & statuses:
1)      Auto calibration
·         Tried another non-linear refinement of autocalibration results (according to “Multiple View Geometry”). It doesn’t improve results significantly.
·         Implemented autocalibration algorithm described in “Untwisting a projective reconstruction” by D. Nister (2004). It gives visually satisfactory results on almost all Oxford datasets (projective reconstructions were obtained via modifying ground truth metric reconstructions). Focal length accuracy is about 10..15% (for any reasonable start guess). On the “corridor” dataset this method fails.
·         Started investigation MATLAB implementation of autocalibration algorithm described in “Globally Optimal Af?ne and Metric Upgrades in Strati?ed Autocalibration” by M. Chandraker et al. (2007).
2)      PCL
·         Did some profiling and performance bottleneck investigations for exact knnSearchBatch for GPU. Main bottleneck is sorting with is expensive for thread and warp.
·         After talk with Radu decided to freeze exact knnSearch development until I have good idea how to make it faster. Current implementation of knnSearch is 30x slower than CPU’s. But radiusSearch and approxNearestSearch are 20x and 50x faster respectively.
·         Learned some of 3D features for point clouds and PCL interface/code for this. Started implementation of PFH, FPFH feature calculation.
3)      OpenCV GPU
·         Optimized some primitive functions (threshold, etc.) as far as users from Yahoo groups complain for low speed-up. (It is really not high for small and average resolutions, because CPU version is super-fast and works several milliseconds).
o   Added TransformFunctorTraits in order to manage transform function behavior for different functors.
o   Optimized some primitives functions that use transform (20-30% speedup).
o   Optimized some of color conversions for cvtColor (30-50% speedup). And fixed minor bug.
·         Discussed how to implement optimized version pyrUp, pyrDown for GPU, but have not started it yet. Hope it will be 4x faster.
Plans:
·         Finish PFH, FPFH, finish test for this, profile and check performance.
o   Possible I need to write different version of kernels for cases: many/average/few neighbors.
·         Evaluate Chandraker’s autocalibration algorithm on synthetic and real datasets.
·         Implement optimized version pyrUp, pyrDown and use it in stitching on GPU for 12Mpix images.
Other:
·         BTW, someone suggested   background-foreground detection   on GPU in our Feature Request Form. This is one point from our plan.

Gary

Radu

Vincent

Alexander

Vadim

• Email Kurt about bundle adjustment approaches

Victor

Kurt

Ethan

James

From last time

Gary

• (./) Talk with Nathan about rsync ’ing opencv.itseez.com to opencv.willowgarage.com/documentation
• Get Ethan to file a bug report on openni drivers
• Have Vincent talk sfm with Victor et al

Radu

Vincent

Alexander

Vadim

Victor

• Write Stephan and CC the rest about refactoring the linemod code

Kurt

Ethan

James

Agenda

• Bundle
• Line mod sync up
  • Transparency
  • Pose
• Model capture
• Tracking

Minutes

• Linemod
  • API and algorithms
  • Separate out pyramid scales
  • Suited for tracking, not so good for recognition
  • We have set up a linemod google doc to coordinate
  • Keep pose + point cloud (train and test). Then use ICP with 3D to 3D fitting
• Model capture
  • Ethan’s method
    • 2 dot patterns, one inverted from another
    • Track’s poses
    • Stores just the ones that are far enough from each other so that we don’t flood the bags
    • These go into a database
    • Can then use the db for model creation
      • Use targets and merges the points together using Mike Krainian’s surfals which gets rid of small noise and it gives nice aligned normals
      • This is then run through meshlab using Poisson to get a mesh
      • Problems: Doesn’t take into account the object itself
        • DWO slam (uses ICP etc to do alignment beforehand)
        • Poisson tends to over smooth but we use to get water tight meshes. Surfals give better reconstruction
    • Want to get a good pipeline together
    • Maybe use DWO slam and then use sufals prior to
• Transparent
  • Initial view based pose
  • minimize distance between training and test edges
    • But can get ambiguities — need multiple view
• Bundle adjustment. Do like SBA but this is GPL, so rewritten SBA from scratch
  • Our SBA is usable from JAVA and other languages since we avoid callbacks (which make it very hard to use with other languages).
    • PTAM uses SBA
• vslam can work with mono or stereo (has many dependencies … too much code). dwo slam is simpler and assumes 3D data (Nicoli’s code). Does bundle adj based on ICP
• Schedules of trips to US for IROS.

Vadim

# (GSOC)
GSOC 2011 has been finished. Python samples are already in the repository.

And I also committed hybrid tracker by Alexander's student Pushkar. The tracker is based on the  paper
H. Zhou, Y. Yuan, and C. Shi,   Object tracking using sift features and mean shift,   Computer Vision and Image Understanding, vol. 113, no. 3, pp. 345 352, 2009.
Video of the OpenCV implementation of the hybrid tracker is here: http://www.youtube.com/watch?v=gIcCDG_Qq6w

#  (Sparse Bundle Adjustment)
Finished SBA implementation; started transforming findFundamentalMat(), findHomography() and calibrateCamera() in order to employ this new engine and debug it (obviously, in those functions the geometry is fixed). The next step would be to reproduce the PTAM tracker.

Victor

Model capture: Implemented an automatic pipeline for model capturing using meshlabserver and meshlab scripts. Different objects require various parameters for surface reconstruction, so I plan to run some scripts with different parameters and choose the best model manually.  Also sometimes meshlab throws different exceptions ant its behavior changes from run to run. An example of automatically created mesh is attached (msvc.png, msvc_model.png). Pose estimation with a grid of circles was integrated into dwo-slam as an initial approximation of the transformation between frames. It helps with poorly aligned parts of an object in some cases.

Untextured objects: pose estimation has been added to linemod.
- Training was modified to store the object point cloud for each template of the train base.
- Implemented pose estimation: having a patch in test image for detected object we assume that the test point cloud for this patch corresponds to the train point cloud of template and calculate rotation and translation between test and train clouds (3D to 3D).
- Results of pose estimation (3D to 3D) on two test images are in the attachment (linemod_pose_1.png). The image shows a test point cloud together with training point clouds corresponding to the recognized objects with pose calculated from linemod.

Transparent objects:
- Reduced noise in the created model of an object by applying k-partite matching. It removes spurious edgels on contours and this is important for accurate pose refinement by edges. See the model of an object before k-parite matching (beforeKPM.png) and after (afterKPM.png).
- Implemented an automatic system for evaluating pose estimation algorithm, now we have a dataset of 8 transparent objects. The attached graph (results.png) shows the percentage of successful refinements (refinement is successful if the final pose is not more than 15 degrees and 3 cm away). The main problem of the current algorithm is convergence to wrong poses which fit test edges well. For example, see a test image (testImage.png) and a refined pose (refinedPose.png). Edges are fitted well but it is the wrong pose (see pointCloudPose.png, the view from aside, green is the table and red is the object). This problem is similar to a PnP problem for an object with linear size much smaller than the distance to the camera. It is hard to solve it without additional assumptions.

Anatoly (GPU)

1)      Auto calibration
·         Tried M. Chandraker’s method. It works on his datasets and mine synthetic dataset. But the method fails on Oxford datasets, even on reconstructions from Oxford page. Spent some time trying to resolve this issue — no success. Author noted that the method isn’t robust and the conditioning of the involved solvers is numerically sensitive. Tried data normalization — didn’t help. Perhaps the root cause is in the presence of noise/outliers. Deeper investigation is required.
·         Implemented projective factorization algorithm from “Multiple View Geometry”. If we use keypoints/matches from Oxford dataset it provides reconstruction quality high enough to get autocalibration accuracy in <20% of focal length via D. Nister algorithm (it works like it was described in the prev. report, but it continue fails on the “corridor” dataset).
·         Replaced the whole reconstruction step with projective factorization based code, here are autocalibration results (principal point and skew are assumed to be known):
Wedham College

Oxford

1148,98

1135,24

537,27

393,08

0

result

1084,27

1083,58

538,98

400,72

-0,38

error

	
63%
	
55%

0.32%

1.94%

0,38

Merton College II

Oxford

1157,69

1163,53

471,91

350,56

0

result

1132,65

1129,59

464,61

356,98

1,14

error
	
16%
	
92%

1.55%

1.83%

1,14

Merton College III

Oxford

1032,57

1043,9

555,53

394,64

0

result

1216,54

1217,04

561,26

399,67

1,88

error

1# 82%

1# 59%

1.03%

1.27%

1,88

focal x

focal y

p.p. X

p.p. Y

skew

This is the best results on real datasets. This method requires that all feature are visible from each frame in dataset.
2)      PCL
·         Implemented simplest version of Point Features Histograms (PFH) and Fast PFH (FPFH) at low level (i.e. they do receive nn-indices, but don’t compute inside). They are not well-tested.
o   Implemented simple tests. Debugging the features now. Learning visualization capabilities in PCL at the same time.
o   Started looking in normal estimation. Going implement it for my interface.
o   Issue: can’t profile my code if OctreeGPU is used for neighbors search before. CUDA Profiler reports an error.
·         Thinking of PCL GPU design in common with accounting Channel conception in PCL# 0.  Reorganized and refactored GPU code, submitted draft to PCL# 0-draft branch.
o   GPU containers moved to separate pcl_gpu_contaners module. Containers interface refactored, simplified and documented (doxigen).  It’s possible to compile pcl_gpu_containers with/without CUDA (just error at runtime)  => PCL# 0/core can use this module in ChannelFactory, for example, if it is necessary (another design approach:  ChannelGPU are constructed only in GPU part).
o   Added pcl_gpu_utils (device layer) module and implemented/moved (from Octree) some device function there for future reuse. Plans to extract/implement more functionality there.
o    Simplified interface of Octree Gpu.
3)      OpenCV GPU
·         Resolved ticket #134# Re-implemented result normalization in gpu::matchTemplate to make sure it handles degenerate cases like the CPU version.
·         Optimized pyrDown and pyrUp (now it works 17-25x faster than CPU version). Now can use it in opencv_stitching to accelerate high resolution panorama rendering with GPU.
·         Refactored BruteForceMatcherGPU (moved some utility functions to device layer).
·         Optimized memory usage in BruteForceMatcherGPU::knnMatch when k==#
·         Updated docs accordingly.
Gary suggested implementation pyrUp/pyrDown with non-power of 2 last week.  Simple blur/resize-based implementation can be done very quickly, but will it give good speed-up?!

Need to think about it. Will put in our plan.
Plans:

1)      Auto-calibration
·         Test autocalibration pipeline on other Oxford datasets.
·         Try feed new reconstruction pipeline output to M. Chandraker’s method.
·         Refactor code, remove obsolete code.
2)      PCL
·         Normal estimation on GPU.
·         Profiling, testing, optimization features. After it, implement high level classes (interface close to CPU).
·         Implementation of repack utilities (SOA <→ AOS, pcl::Normal → float3 or float4]) because such repacks are faster on GPU than on CPU. And will be very useful in future.
3)      OpenCV GPU
·         Update opencv_stitching with new gpu functions (knnMatch, pyrUp, pyrDown). Add support of different types and interpolations modes to gpu::remap (it is also used in stitching).
·         Implement separable filters for big kernel size (ticket #1134). Now we support ksize <= 16 only.
Other:

·         New user suggestion:   Random Trees on GPU    by PhD student from BMW group at Munich (in our Feature Request Form).

Gary

Radu

Vincent

Alexander

Vadim

• Email Kurt about bundle adjustment approaches

Victor

• Sync up on model capture with Ethan
  • Ask Alexander

Kurt

• Send Victor what Hinterstoisser is doing with model capture
• Help Victor sync up on model capture
• Send pointers to Ecto

Ethan

• Sync up with Victor (DWO slam) on model capture
• Send Victor pointers

James

From last time

Gary

• (./) Talk with Nathan about rsync ’ing opencv.itseez.com to opencv.willowgarage.com/documentation
• Get Ethan to file a bug report on openni drivers
• Have Vincent talk sfm with Victor et al

Radu

Vincent

Alexander

Vadim

Victor

• Write Stephan and CC the rest about refactoring the linemod code

Kurt

Ethan

James

Agenda

• Model capture
  • Hinterstoisser’s registered 2D codes
  • Ethan’s dots and inverted dots
  • Victor’s dwoslam
• LineMod
  • Is this API OK?
• Transparent items
• Next in object rec
• IPhone port
• How to make LatentSVM

Minutes

• Model capture
  • Alexander Shishkov sent Ethan an email about how dwo slam works. Alexander is on vacation this week
    • See email ‘’"code for model capturing with dwoslam"’’
• Linemode
  • Pose refinement … cohere into a toolbox?
    • Solve PnP
    • LePetit’s — has code online
    • Version with Ransac
    • edge refinement … toolbox in ROS for doing this
• Transparent object
  • edge refinement/pose refinement using Nlopt
    • Has clutter problems, but if it is in weak clutter, pose is good
• Iphone port
  • Almost done
  • needs tutorial on how to write an app with opencv on iPhone
  • works with simulator
  • Need to register as Apple developers
• Sparse bundle adjustment
  • Works well on test datasets sba package.
  • Extending to unknown cameras, PTAM like functionality.
• Phase correlation is in trunk with docs, tests, sample code. Thanks to Will Lukas!

Victor

Untextured objects:
The format of the linemod storage was improved. All trained templates for all object classes were saved in one xml-file before. Now templates for each class are saved in their own folders. It allows to run the algorithm with a subset of train objects.  It also eliminates the problem of very large xml file reading.
Evaluation of pose quality was implemented. Currently it does not take into account the symmetry of objects (11 from 12 objects of our base have symmetry), so two poses that differ by a rotation around symmetry axis are considered different.
Analyzed linemod interface proposed by Patrick. It looks fine at the first reading except that trained templates for all classes are stored in the same xml-file.

Transparent objects:
Oriented chamfer matching was adapted for work with contours of the object. It increased ratio of successful pose refinements on ~10% on average.
Multiple cameras were used in pose refinement by creating a stereo pair of EOS 40D cameras. Multiple images reduced ambiguity of poses (see an example in attachments). Currently the main problem is ambiguity between correct pose and upside-down pose because objects are nearly cylindrical.

OpenCV on iOS: A sample by Xiaochao was reproduced on iPhone simulator.

Other:
Alexander is on vacation this week.

Attachments:
linemod:
linemodPoses.png -- examples of poses returned by linemod.
transparent objects:
init.jpg -- initial pose
left.jpg -- refinement by the left camera alone
right.jpg -- refinement by the right camera alone
both.jpg -- refinement by using both cameras

Vadim

# (Contributions)
Will Lucas contributed phaseCorrelate() function: http://en.wikipedia.org/wiki/Phase_correlation

#  (GSoC)
While GSoC 2011 is officially finished, Alexander is still working on Python bindings and the samples. He has added wrappers for the feature detection and descriptor matching framework from features2d module.

#  (Sparse Bundle Adjustment)
The simplest case (calibrated camera, compensated distortion) has been debugged on couple of test cases from SBA package. Now the code is being extended to support unknown camera matrix and distortion parameters in order to calibrate camera in parallel with the geometry reconstruction.

Anatoly (GPU)

Accomplishments & statuses:
1)      Auto calibration
·         Tested current camera autocalibration pipeline on other Oxford datasets and wrote short summary.
·         Tested projective factorization algorithm with M. Chandraker’s autocalib method — it still provides poor results.
2)      Opencv GPU
·         Fixed tickets: #1344 (compilation), #1353(minor bug in CudaMem::createMatHeader), #1323 (small image size checks for SURF_GPU)
·         Implemented gpu::remap for all types & flags. (we need it for stitching on GPU)
o   Added BORDER_REFLECT and BORDER_WRAP extrapolation support GPU module.
o   Added Filter Functors (nearest, bilinear) to device layer.
·         Found 1st NVCC bug, investigated, created simple repro-case and submitted it (id872731).
3)      Opencv stitching
·         Turned opencv_stitching application into OpenCV module + sample. Started simplifying stitching interface for end users. Going to speed 3 days for this. [Tegra team request. They’ve got a lot of user requests on their forum].
·         Investigated opencv_stitching failure on photos taken from Android. The reason is wave correction step (that just shifts panorama before rendering): If mobile photos are taken with portrait orientation → vertical straightening is required. But the step tries to make horizontal panorama, and as result it returns bad rendering position on sphere. This was repaired by disabling wave correction by default. Probably will implement vertical wave correction step.
·         Optimized blending & panorama rendering with to gpu::remap/gpu::pyrDown/gpu::pyrDown in opencv_stitching. Whole stitching pipeline is ~ 3x faster now for 12MPix images.
4)      PCL
·         Implemented and debugged normal estimation on GPU for non-organized point clouds. Simple optimization iteration. Guess can make faster, want to see its fraction of time in ComputerNormal – ComputerFeatures pipeline before.
·         Found 2nd NVCC bug during normal estimation debugging. Simple repro-case creation is very hard. Guess it’s similar to bug in NPP_ Staging::IntegralImages. At least workaround is the same   extra _threadfence_block() intrinsic.
·         Debugged FPFH in order to reach results similarity with CPU code.
a)      Found that in CPU code: FPFH = sum(SPFH * Wi), where Wi   sqrt_distance, but according to manual here, it should be distance (not sqr).
It is a bug in PCL?

b)      Despite of replacing   fast math   intrinsic with full precision functions (sqrt, atan2, cos, sin, etc), I can’t reach byte-wise similarity between CPU & GPU. The same point can vote in different neighboring bins on CPU and on GPU (different computational error).
·         Started debugging PFH.
Plans:

·         Refactor stitching module. Add tests and docs. Fix wave correction step in stitching sample. Add support of rotation initial guess into stitching sample (example, from compass on Android device).
·         Implement separable filters for big kernel size (ticket #1134). Now we support ksize <= 16 only.
·         PCL: profiling, testing, optimization features. After it, implement high level classes (interface close to CPU).
·         Implementation of repack utilities for PCL because such repacks are faster on GPU than on CPU. And will be very useful in future.

Action Items

Gary

• See where Ethan is on dwo slam from Alexander
• Talk to Marius about scaling linemod using wta coding
• Try to find transparent object recognition paper(s)
• Check with Suzanne about intern house for Victor and Vadim

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

James

From last time

Gary

Radu

Vincent

Alexander

Vadim

• Email Kurt about bundle adjustment approaches

Victor

• Sync up on model capture with Ethan
  • Ask Alexander

Kurt

• Send Victor what Hinterstoisser is doing with model capture
• Help Victor sync up on model capture
• Send pointers to Ecto

Ethan

• Sync up with Victor (DWO slam) on model capture
• Send Victor pointers

James

Agenda

• No real meeting today, Vadim is onsite, Victor just left
• SLAM dwo
• Latent SVM: NNSU Felzenszwalb team about adding docs, samples, tests and TRAINED classifiers
• Look into putting back documentation comments
• make it clear how to get or not get opencv_extra

Minutes

• No official meeting today, just notes. Vadim is onsite and Victor just left.
• Vadim and Gary are working on improvements to the ORB descriptor, basically
  • Making FAST even faster and maybe innately more resistant to finding edges TBD. We also may work on repeatability … but not at the cost of making FAST SLOW.
  • Using ideas from Google’s (‘’Yagnik, Strelow, Ross, Lin’‘s) Power of Comparative Reasoning paper to boost ORB’s discriminability .
  • We may also use separable Gaussian blur instead of integral images which may help with rotation tolerance TBD
  • It is obvious, that since we estimate moments at corners anyhow, these could be used to do a full affine instead of just rotation correction. Trades off with speed, but might be worth it.
• Work to scale Linemod is underway, Gary reimplemented to whole algorithm to make it easier to experiment with.
  • Unclear as yet whether scaling (by which I mean slower than linear slow down with increasing numbers of object learned) will work with these kinds of sparse features.
  • LSH and NN type methods are dubious.
  • Will use the kind of pyramid tree that seemed to work with BiGG.
• There will be a push for transparent objects as is “apparent” from these recent meeting notes.
  • Gary is going to add some gradient methods late this week/early next to add to the NAN and pose fitting.
  • This will be followed by some as yet secret sauce to make transparent object recognition and pose reliable.
  • Possibly see RSS which is the target for this effort.
• TOD underwent major improvements thanks to Ethan and Vincent — it is now very stable in pose recognition and much more able to find objects even with the terrible kinect RGB camera. Speeding it up and getting it on the robot for grasping is this week’s goal.
• Trained models for LatentSVM are now in opencv_extras. Documentation is restored. Work is underway to get the cascade version in. Hope to optimize this — it may be good for general objects such as chairs, shelves, etc

Victor

TRANSPARENT OBJECT POSE:
Created a new dataset with 5 transparent objects captured by the calibrated Kinect and the EOS 40D stereo pair. Kinect is used as a glass detector:
# The approximate object contour is extracted from a Kinect depth map by using its NaN values.
#  This contour is refined using the Kinect RGB image by applying the GrabCut algorithm. OpenCV snakes and GrabCut were compared on several images for this task (for example see contours.png) -- GrabCut is the winner.
#  Our pose estimation algorithm is run using the refined contour as test edges and the computed pose is returned as the initial pose which is refined using the stereo data.
Sometimes the algorithm returns an upside-down pose instead of the correct one. To avoid such problems we added an additional optimization step that uses the upside-down pose as an initial pose. The pose is estimated correctly in 79% of cases. The table below shows pose errors for each of the objects. This error includes the constant displacement between the training and test object poses, a more precise measurement is on the way. [Ilya]

----------------------------------------------
object      std dev of        std dev of
rotation (rad)    translation (m)
----------------------------------------------
bank         0.0744             0.0083
bottle       0.0787             0.0131
bucket       0.0855             0.0104
glass        0.0723             0.0058
wineglass    0.0905             0.0117
----------------------------------------------
mean         0.0803             0.0099

MODEL CAPTURE:
The first implementation of ICP-tracker based on the paper of Kok-Lim Low "Linear least-squares optimization for point-to-plane ICP surface registration" (and partially on the paper of Yang Chen "Object modeling by registration of multiple range images") was finished. Some experiments on comparison the implemented ICP-tracker, dwo-slam and a method based on fiducial markers were made to choose more stable approach. Test system was implemented to evaluate R, t obtained by the ICP-tracker and dwo-slam. Quality was evaluated on the basis of the method using fiducial markers (circles grid) due to its better stability. So far ICP tracker shows worse performance (33% of large errors in pose compared against 6% from dwo-slam). Further experiments with ICP tracker (in particular, parameter tuning) are on the way. [Alexander]

LATENT SVM:
The documentation for C interfaces of latent SVM was restored from SVN history (it was lost in updating OpenCV documentation). Converted it to rst format (from LaTeX). The trained VOC2007 database models and the conversion script from Felzenszwalb's format to OpenCV's were added to opencv_extra/testdata/cv/latentsvmdetector/.

Implemented C++ interface wrapper for latent SVM detector (cv::LatentSvmDetector). Added support of several models (C interface doesn't support this feature). Implemented sample for it in /samples/cpp/latentsvm_multidetect.cpp. Tests and documentation for cv::LatentSvmDetector methods were added. [*Maria*]

LINEMOD:
Moving our applications (for training and evaluation linemod) to the refactored linemod implementation is in progress. [*Maria*]

OPENCV:
- A chapter on a cascade classifier training was added to OpenCV user guide [http://docs.opencv.org/trunk/doc/user_guide/ug_traincascade.html]. [*Maria*]
- Updated Buildbot to 0.8.5 version. Fixed problems with svn synchronization on MacOS.
- Fixed #1030, #1319, #1330, #1372, #1392, #1406 tickets.
- Fixed bug with using external ZLIB library (rev. 6829).
- Modified Buildbot scripts, now the latest installers for OpenCV are copied to our file-storage nightly. [Alex, Andrey]

OTHER:
Maria was on a sick leave on Thursday (October 6).

Vadim

# (Feature Detectors)
FAST detector has been optimized with ~# 5x speed up (depending on the image and the parameters). The new implementation is also much more compact and easier to extend with more complex tests on the corners (as there is no hard-coded decision tree).

#  (Feature Descriptors)
Extended Hamming distance has been added in the case when the elements of a descriptor may take values other than 0, 1 (so that they take more than 1 bit each). This is to extend ORB descriptor using "WTA (winner-takes-all) hash" technique.

Anatoly Baksheev

1)      Video stitching demo
·         Prepared beta1 and beta2 of the demo. Implemented calibration helper (scripts + utils). Prepared detailed readme.txt.
·         Integrated GPU face detector into the RT stitcher demo. Implemented naive seam correction to placing seam through faces.
·         Implemented estimation  of projection plane distance (at faces). Tried to triangulate face centers using them as pairwise matches. Rays going from cameras through face centers are almost parallel, so distance can’t be estimated accurately enough. Even averaging over all image pairs doesn’t help. Implemented experimental version of gradient descent method for projection plane distance estimation, where cost function is the sum of distances between warped face centers.
·         Integrated multi-band blending to the pipeline.
As result we have beta2 and recorded video that demonstrates how algorithm works.
Issues:

·         Performance  of RT stitcher with smart seams and plane estimation is about 1..2 FPS.
·         Demo supports only one face.
2)      PCL
·         Implemented and debugged PPF, PPFRGB, PPFRGBRegion,  PrincipalCurvatures. PFHRGB. Tested similarity with CPU results. Features returned are very close. No ideas for optimization. Can commit this to public repository right now.
·         Implemented but not well tested VFH. Going to commit after testing.
·         Started reading Kinnect Fusion paper, learning ICP, etc.
3)      OpenCV GPU module
·         Implemented brute force approach for cv::gpu::convolve for small kernel sizes.
·         Implemented OpenCV-like API (namedWindow/imshow/pointCloudShow/waitKey) in our GLUT-based GUI library (highgui_gpu, we use it for demos). That speed-ups development and debugging demos    we can display intermediate results using OpenGL by one string of code.  Now it is separate from OpenCV, but It can be converted to OpenCV module very easily.
·         Fixed bug in BROX (appeared after our fixes, reported by Anton).
Plans:

1)      Learning Kinnect-based 3D reconstruction from the paper.  Seminar at Itseez.
2)      Investigate convolve performance for Fermi on different datasets. Final comparison of updated functions with Libjacket.
3)      Possibly create wrapper for Brox.
4)      Discussions about unified command line interface for samples.

Gary

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

James

From last time

Gary

• (./) Figure out what Vadim/Victor should work on while here

Radu

Vincent

Alexander

Vadim

• (./) Edit the download guide to make it clear how to get or not get opencv_extra
• Talk with NNSU Felzenszwalb team about adding docs, samples, tests and TRAINED classifiers
• Look into putting back documentation comments

Victor

Kurt

Ethan

James

Agenda

• Travel dates
• Improving calibration
• Khronos
• Registration code
• Iphone port
• Better website

Minutes

• Travel.
  • VE: Tue→SF→AR Until Thrus.→Mon (26). Sept 30→Oct 8
  • VP: Oct 4→16
  • Vadim on vacation.
• Xilink may fund fpga interns. Could be used for
  • Camera-Projector combos
    • Point cloud massage from
  • Close range depth sensing
  • Cameras with basic feature pre-processing such as linemod. Daisy? HoG?
  • Cameras with basic pattern or event detection build in. Tracking.
• Khronos
  • We have another rev of the proposal. Moving forward.
  • Need to decide in what form we will join Khronos? As OpenCV.org? Perception.org?
• Better website. We are overdue for an overhaul.
• Dot pattern calibration
  • Ethan will send his VTK calibration renderer for helping develop the calibration pattern.
  • This will be a great general utility
• Iphone OpenCV port compiles, not independently tested on iPhone yet.

Anatoly (GPU)

Accomplishments & statuses:
1)      Opencv GPU
·         Added performance tests for gpu module. This tests are intended to track performance difference between revisions. If after our fix (refactoring) something become worse we catch it.
·         Added bicubic filter to device layer.
·         Fixed bug in gpu::remap on win32 (workaround for nvcc bug). Fixed bug in stitching in GpuSurfFeaturesFinder (missing upright parameter).
·         Implemented gpu::resize for all types & flags.
2)      Opencv stitching
·         First iteration of stitching module refactoring (moved all into detail namespace, added simple class Stitcher, added sample).
·         Spend some time to discussions of stitching API with Tegra team. Communicated with Peter Karasev. He’s going to contribute stitching sample which can handle rotation initial guess / timestamps.
·         Fixed link bug in stitching module under Windows.
3)      Video Stitching Demo
·         Created demo app which stitches video from fixed cameras. Several iteration of its optimization. As result blending works up-to 80fps on 3 Full HD images.
·         Tested on web cameras, found poor quality on web-cameras. Investigating.
4)      PCL
·         Found and fixed 2 bugs on gpu::Octree & FPFH. Finished debugging PFH.
·         Implemented high-level classes for Normal estimation (indices & surface support). Added indices support to OctreeGPU:: radiusSearch.
·         I am near to finish high-level classes for PHF & FPFH. Just unplanned and urgent video stitching demo took some time. After going to compare performance: CPU [F]PFH + Kdtree radius search vs.at