Anatoly (GPU)

Accomplishments & statuses:
1)      OpenCV GPU Farneback optical flow
·         Experimented with different approaches of porting the Farneback’s optical flow algorithm on GPU. Now it works about 8-15x faster than the CPU version on 640×480 images depending on parameters. Refactored the GPU optical flow code a lot.
·         Investigated difference between GPU and CPU optical flow results: it seems the reason is in difference of resize() function. Implemented purely texture based GPU analog with the bilinear interpolation (built-in one) – it didn’t help.
Issue:
·         With Gaussian kernel error difference between CPU and GPU is always higher then with Box filter kernel.
2)      OpenCV GPU pyramidal Lucas–Kanade optical flow
·         Finished with GoodFeaturesToTrack detector and Pyramidal Lucas Kanade (Sparse + Dense) and integrated it to OpenCV GPU module.
·         The speed-ups below are archived:
GoodFeaturesToTrackDetector:   6-8x faster than OpenCV CPU
PyrLKOpticalFlow Sparse:  1.5-2.1x faster than OpenCV CPU (TBB + SSE).
PyrLKOpticalFlow Dense:   2-2.5x faster than BroxOpticalFlowGPU contributed by NVidia.
Test system was Tesla C2050, Core i5-760 2.8Ghz (quad core). OpenCV CPU version is optimized extremely well. For one pyramid level, our implementation is 0-20% faster than CUVILib’s not pyramidal LK implementation.
3)      PCL. No significant updates.
·         Experimented with RGBD benchmark. Adjusting volume pose relative to scene, increasing ICP iterations numbers and tweaking other parameters fixes misregistration on the dataset. Committed minor fix for normals computation from tsdf (made them smoother), it seems it improves tracking quality according to the tests.
Plans:
·         Investigate why the Gaussian-based version of Farneback differs so much from the CPU. Make the final refactoring and commit the code. Document for all optical flow algorithms. Investigate resize implementation (it differs too much from CPU).
·         Probably start adding NPP 4.1 functions to OpenCV or working on background substraction.
·         Continue with ACM Queue article.

Action Items

Gary

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

From Last Time

Gary

• Start functionality list

Radu

Vincent

Alexander

Vadim

• figure out 2D Barcode functions

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Release
• CVPR Workshop
• GSOC’12
• Khronos
• Tracking with known camera movement
• FPGA discussion

Minutes

Transparent objects [Ilya]

Finished the implementation of the paper "A Geodesic Active Contour Framework for Finding Glass" by K. McHenry and J. Ponce, CVPR 2006.
Added the texture distortion feature by using textons from the paper ”Contour and Texture Analysis for Image Segmentation” by J. Malik, S. Belongie, T. Leung, J. Shi, ICCV 2001.
Improved SVM accuracy by following A Practical Guide to Support Vector Classification from the LIBSVM.
Post-processed results of the Felzenszwalb's segmentation algorithm to remove thin regions on boundaries of objects.
The attached animated GIF shows a Geodesic Active Contour evolution on a training image computed with the implemented algorithm.

RGB-D visual odometry [Maria]

Made a lot of experiments with ICP, RGBD and different versions of merged ICP+RGBD odometry on freiburg1_desk http://cvpr.in.tum.de/data/datasets/rgbd-dataset/download#freiburg1_desk mainly. KinFu ICP is more stable and doesn't break trajectory critically on the data. RGBD is sensitive to the following problems with rgb and depth data:
- Due to insufficient synchronization between rgb and depth data, motion blur, ... 3d points don't have their real colors.
- Matched frames can be far from each other.
In such cases RGBD gives a pose far from ground truth. So RGBD breaks the trajectory several times, but it has less than ICP median drifts of translation and rotation per frame for well synchronized data (see previous report). Tried several approaches to merge ICP and RGBD. Found that more stable version of merging is the following. RGBD pose is used only if both points error and photo consistency error are less than the same ones for ICP pose. But sometimes this merging strategy gives a trajectory that has larger absolute error than ICP.

OpenCV tests and bug fixing [Alexander Reshetnikov]

Applied some solutions for fixing the bugs associated with frame count computing and video clip positioning (feature #850 and report from the ticket #120).

Results of frame count and positioning tests (default testing infrastructure is Ubuntu x64, actual frame count is 125):

Closed tickets # 850, 1154.

OpenCV infrastructure [Alexander Shishkov]

- conversation with hetzner.de support about upload speed on our server.
- modified Apache settings for OpenCV SVN server; now full checkout of OpenCV trunk (about 1GB) from office takes about 10 minutes
- modified UUID on local SVN mirror, so it is possible to do relocation from local mirror to the public repository, migrated local mirror to Apache, forbade commits to the local mirror
- updated wiki on Buildbot project (#1073), updated Android NDK on Ubuntu slave (#1245), added link to the trunk documentation to the Buildbot main page(#1239), added highgui tests tot the Android builder(#1244)

Other:

- fixed ## 1599, 1602, 1604, 1363, 1492, 1339, 1614
- started the work on FFMPEG problems ## 1615, 924

Anatoly

Accomplishments & statuses:
1)      Optical Flow
·         Revealed and fixed the bug in the high level Farnebeck GPU function which led to difference between CPU and GPU results.  Found out that cv::GuassianBlur() on CPU uses some hard-coded sigma values in case of small kernel sizes. That also caused difference between the CPU and GPU. Ensured the GPU version behaves the same way.
·         Integrated Farneback GPU to OpenCV. It works 13..22x faster than the CPU version (Intel Core i3 vs. NVIDIA GTX 470) on 640×480 size images.
·         Miscellaneous:
o   Added accuracy tests, performance tests, docs.
o   Created sample, updated performance sample (samples are convinced as API example for developers)
o   Updated demo pack with the optical flows. (Demo pack was convinced as use-case examples for users).
2)      OpenCV GPU
·         Revealed incorrect behavior of linear filters for kernel sizes >=11×11. Fixed it so that it supports kernel’s size up to 32, also new implementation is a bit faster.
·         Added support of different border extrapolation modes into my implementation of Gaussian blur — to be similar to the CPU version.
·         Created wrappers for new NPP arithmetic only functions.
3)      Other
·         Prepared ACM article draft version. Guess after adding Khronos process it will reach recommended size.
·         Hiring activities. I never put this in reports, but I spend time for participating interview for Tegra and so on.
·         Started learning video stabilization theory.
4)      PCL
·         Fixed the BUG in marching cubes in Kinect Fusion
·         Minor compilation fixes and tests for different platforms (like C++0x flag in CMake breaks NVCC compilation and so on)
·         Started learning code and papers for future people detection GTC demo.
Plans:

·         Adding NPP 4.1 functions and GPU video decoding.
·         Continue learning video stabilization theory.
·         ACM Queue article
·         Continue with pcl_people on GPU.

Action Items

Gary

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

From Last Time

Gary

• Start functionality list

Radu

Vincent

Alexander

Vadim

• figure out 2D Barcode functions

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

“OpenCV submission to Google Summer of Code”: http://code.opencv.org/projects/gsoc2012/wiki/Gsoc2012

• GSOC org application has started (Gary)
• Khronos
• CVPR Workshop prep (Victor)
• Web page for OpenCV.org (Vadim)
• Release
• Nonprofit
• Transparent
• Bar codes (James is limited 2 string length 3?)
• Xilix/Zync

Minutes

• Wiki is on code.opencv.org
• GSOC Ideas page is set
  • OpenCL Optmization:
    • We have contributors already, get one or more of them to be GSoC students
  • Mobile vision app development:
    • Vision apps on phone and tablet. We want a very clear “hello world” type of example to make it easier for the rest to start developing mobile vision applications. People can apply for:
      • iOS (iPhone) examples
      • Android example
  • 1D and 2D Barcode readers:
    • We have a basic datamatrix reader. Expand to other kinds of barcodes (1D, other 2D).
  • Natural barcodes:
    • OpenCV can read calibration patterns. Calibrate a camera, rectify it’s output. Put a picture next to a calibration pattern in a scene. Find 2D interest points with features2D. Measure them in the image using the calibration pattern. Use the pattern of interest points and their measures to turn the picture into a calibration pattern/barcode so that the computer recognizes the picture and knows where it is in relation to the iamge.
      • Stretch goal, embeded and retrieve watermarks in the image.
  • Python example:
    • We’re finding the python interface extremely useful and productive. It needs a lot more examples
  • Tutorials:
    • Continue with OpenCV Tutorials
  • Inverted index:
    • Help create an inverted index of functionality to functions. That is, say, edge detection would list Sobel, Scharr, Canny etc.
  • Train Classifiers:
    • OpenCV has many classifiers. Train them carefully on useful objects (faces, people, cars, bikes, plates, cups, bowls etc) and we’ll allow others to use those trained categories.
  • Structure from Motion (SfM):
    • Help adapt libmv for OpenCV.
  • Course-ware:
    • We want to start offering vision courses using OpenCV.
      • If you are expert in a topic (say it’s your area of graduate research, or you’ve just finished a project or course in a particular topic and can explain it well)
      • Write up the topic, the math and then develop a step by step example of it using OpenCV.
• Khronos meetings
  • Victor and Vadim attend phone meetings. Gary may start.
  • Mostly it’s covering SOW philosophy
• Vincent has ideas for opencv.org (Alexander)
  • More tutorials etc
  • Want to know what’s happening.
    • Clear roadmap
    • Change logs even in trunk
  • Clearer description of modules, what it can do
  • Small jobs that others can do
  • Brainstorm: Website where people can share ideas of what people want/votes
  • Show that project is active. Show for example a page of sponsors
  • Forum for developers only (make a tab on the website: contacts, developers/users more prominent)
  • Improve docs and tutorials obviously
  • Official contact when there is a non-profit
  • Want levels 1 is core (image processing) 2 tracking … 3 applications
  • Inverted index
• Release: Beta eo March, final release eo April
  • ROS freeze, March 15
• lvm (lib virtual machine)
  • C++ running natively in web browser
  • people use for OpenCL
• Transparent
  • Working on segmentation
    • “Perceiving light versus material” by F.A.A. Kingdom (Vision Research, 2008) and “Low-level image cues in the perception of translucent materials” by R.W. Fleming and H.H. Bulthoff (ACM Transactions on Applied Perception, 2005)
    • Contrast, texture distortion, color distortion
• We have datamatrix barcodes from James based on ideas from libdmtx
• Barcodes
  • We have datamatrix due to James Bowman
  • It would be nice to have
  • circular tag modules
  • Put these in opencv apps
• Think about opencv apps site
• OpenCV with OpenNI driver. Using Asus Pro driver
• Bug: Shape context. Vincent file bug about it.
• Maria on visual optical flow with rgbd
  • Near term goal is to integrate code into OpenCV
    • Even more robust motion estimation with pure translation assumption
    • Robust detection of visual tracking failure
      *

Anatoly (GPU)

Accomplishments & statuses:
1)      OpenCV GPU
·         Finished VideoCapture_GPU implementation under Linux ( but have not integrated in OpenCV)
·         Improved arithm functions dispatching between NPP and our code (~2x faster now).
·         Created GPU path for gpu::PyrLKOpticalFlow (cpu version has strange behavior in error calculation, created ticket #1651).
·         Revealed some inconsistencies in gpu::convolve() for small kernel sizes specializations (ticket #1639). Fixed error in gpu::convolve() docs.
2)      Face detection on GPU. As soon as Anton Obukhov doesn’t longer support FD code, we are going to fix its all crashes for CUDA4.1 by our own.
·          Fixed HaarCascadeLoader test failure.
Active issues:
·         Visualization test fails.
·         Haar classify kernel crash with CUDA 4.1
3)       Video Stabilization.
·         Implemented trimming which ensures that there are no black border artifacts due to stabilization in output video by trying to relax estimated stabilization transformations. It works better than naive ad-hoc which doesn’t stabilize at all when there are artifacts.
·         Implemented Fast Marching Method which is required for in-painting steps. Wrote test comparing it with distanceTransform() (as distance map can be computed via FMM naturally).
·         Implemented color in-painting via averaging of neighbor pixels colors. Implemented motion in-painting which propagates local motion into areas where optical flow isn’t known. Experimented with different optical flow algorithms. As it was mentioned in the original paper motion in-painting isn’t perfect technique and there are some artifacts.
4)      PCL / Human Tracking
·         Euclidian point cloud clusterization. Learned the algorithm and its code, understood bottlenecks. Read several papers about connected components on GPU. No idea how to accelerate the cauterization with GPU. It’s universal and slow (800ms). And even taking into consideration that input cloud is organized doesn’t reduce amount of work to be done. Instead, two lightweight algorithms were implemented on CPU which work 116ms and 14ms (floodfill-like). They produce some approximation of output from original. Experiments showed that 14-ms version is enough and doesn’t break human tracking pipeline. Accelerating this 14ms-version with GPU is also not easy and not reasonable as currently it is not a bottleneck.
·         Seeded hue segmentation. The same complexities as with previous (work time 200-300ms, not idea for GPU). The approximate method used for clusterization doesn’t work here (it cuts persons hands). Accelerated CPU code by refactoring and sampling points a points queue inside (some another kind of approximation), so that it works 70-90ms for middle distance objects. Further optimization is possible with OpenMP, so expect work time to be ~30-40ms. Guess, better results are achievable by developing another smarter (than floodwill) 2D segmentation with depth constraints.
·         smoothLabelImage. It’s a kind of filter. Created GPU implementation for it, so that run time reduced from 83ms to 1ms.
Overall frame time for human tracking was reduced from 2.5sec to 400ms.
Plans:
·         Integrate VideoCapture_GPU to OpenCV. Fix CascadeClassifier kernel crash.  Fix gpu::convolve implementation (ticket #1639).
·         Video stabilization: Work on exceptional cases handling like poor in the sense of number of features datasets and datasets with large moving objects.
·         Implement volume shifting for Kinfu (Kurt’s request) or something other in PCL.
·         ACM Queue article

Ilya

Transparent objects [Ilya]
Studied psychological papers on perception of transparent objects. They are useful for us because they describe heuristics used by people to perceive transparent objects and we can derive features from them for glass segmentation. The most useful papers are: “Perceiving light versus material” by F.A.A. Kingdom (Vision Research, 2008) and “Low-level image cues in the perception of translucent materials” by R.W. Fleming and H.H. Bulthoff (ACM Transactions on Applied Perception, 2005). Added 3 features to our algorithm based on read papers:

1. Added Michelson contrast.

2. Added root mean square contrast.
These two features are discriminative because contrast plays the important role in perceiving glass. Look at two examples (episcotister.png and contrast.png):

(images from the paper “Toward a perceptual theory of transparency” by M. Singh and B.L. Anderson (Psychological Review, 2002))

Regions p and q on the first image look as transparent and they have lower contrast together than a and b. However, if they have bigger contrast (see the second image) then this perception is gone. In fact, the surrounding regions look as a transparent layer with the hole in the middle because they have lower contrast now.
3. Added distance between median colors in regions as a feature because a chromatic change is useful for perception too. Our regions are rather uniform after over-segmentation with Turbopixels so we can use just a median color, not full histograms.
Adapted glass boundaries computation for Kinect data. McHenry et al. classify only adjacent regions as glass-background. But our regions are much smaller so we can expect glass-background pairs to be separated by other regions. Added classification of region pairs in a neighborhood and aggregated results with a voting procedure.
See example of our algorithm working on a test Kinect image (the animated GIF):

(Note the algorithm uses RGB data only that is a depth image is not used for segmentation).
RGB-D odometry [Maria]
Refactored the RGB-D odometry implementation to put it in the OpenCV contrib module.

Found the bug in estimation of time of our RGB-D odometry working. The time was normalized on iteration number after some experiment, but the time from the original paper on RGB-D odometry is for all iterations. So optimized the implementation in more than 10 times. Main optimizations:

1. Switched to the fast Sobel version (16S).

2. Optimized image warping.

3. Chose a subset of textured pixels to estimate a transformation.

Now the implementation works the same time as in the paper (~80ms).

Started experiments on the Kurt’s dataset.
OpenCV infrastructure and issues [Alexander Shishkov]
OpenCV infrastructure:

Investigated the new Buildbot API in 0.8.5.

Began to work on FFMPEG builders for the OpenCV Buildbot.

Updated settings and the theme of the test OSQA installation.

Investigated AskBot, began to install it on our hosting (we plan to compare it with OSQA internally).

Done default settings for the GSOC project on code.opencv.org.

Fixed some problems with the code.opencv.org theme.

Communicated with the designer of the opencv.org website.
OpenCV issues:

Done some infrastructure and documentation issues (##1646, 1645, 1644, 1637, 1618, 1546, 1507,  1477, 1466, 1456, 1405, 1398, 1289, 1260, 521).

Updated png and tiff libraries used for OpenCV compilation on Windows (#1609).
OpenCV tests and bug fixing [Alexander Reshetnikov]
Configured ffmpeg 0.10 on the local machine to write and test the new OpenCV wrapper. It is based on the ffmpeg class written by Andrey Morozov.

Checked the work of the Andrey’s ffmpeg class on frame count and positioning highgui tests:

1. The frame count test.

Results (actual frame count is 125):

Results strongly depend on the file format because for different containers we can find the frame count property in different ffmpeg structures (now there is no universal rule to get it).

2. The positioning test.

The class gives fully correct results on positioning tests. However, the class has some problems with reading some formats (also it depends on a local ffmpeg configuration).

Began to write the new ffmpeg wrapper. Achieved compatibility of this wrapper with the old implementation of the ffmpeg wrapper.

Vadim

Action Items

Gary

• (./) Finish GSoC application http://code.opencv.org/projects/gsoc2012/wiki/Gsoc2012
• Submit ECCV workshop
• Contact James about limitations in 2D barcode
• (./) Ask Kurt about visual odometry 2D+3D

Vincent

• ROS integration with latest possible OpenCV trunk
• Describe OpenNI for ASUS prime sense

Alexander

Vadim

• Talk to Anatoly about stand along kinfu

Victor

Kurt

• Talk to Maria on visual optical flow with rgbd
  • Near term goal is to integrate code into OpenCV
    • Even more robust motion estimation with pure translation assumption
    • Robust detection of visual tracking failure

Ethan

• Get OpenCV for Asus prime sense

From Last Time

Gary

• Work on non-profit
• (./) GSoC application

Vincent

• Assert ownership of direction of Itseez team
• Bug report on samples

Alexander

Vadim

• (./) Get Alexander to set up wiki for OpenCV.org

Victor

• Get on the Mobile workshop prep.

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Python — have a volunteer to work on demos etc
• Vincent ROS integration with latest OpenCV trunk?
• OpenNI for OpenCV trunk
• New dense optical flow looks really good. Congrats to Maria!
• Kinfu integration. IPI has it stand alone now.
  • New optical flow technique means we might not even need kinfu, might have something better?
• Ethan’s contributions
  • Patches for ASUS, really fast normal estimation (~instant even on CPU)
• GSOC — our application is in, we find out what the decision is on Friday at noon, PDT.
  • See our ideas page http://code.opencv.org/projects/gsoc2012/wiki/Gsoc2012
• Khronos updates
• Non-profit progress
• User web page status
• Asus support. Ethan’s contributions (super fast normals)
• Eigen
• Rotations convenience functions
  • Quaternians, Rodriges, R,t and just some common functions we face (segmentations, histograms, morphology)
• Connected components
• Bump hunting (Gary writing) Finding density peaks in data
• ECCV application
• CVPR. Gary’s paper is in. Workshop dev => course ware.
• Transparent

Minutes

• GSoC … find out on this Friday
• Python Abid will start working on some examples. He’s not going in GSoC, we’ll find another for that
• Latest OpenCV in ROS, Vincent to call Ethan on an issue
• Kinfu in OpenCV in GPU version.
• New dense optical flow from Maria is working well, document, samples, great job!
• ASUS divers submitted as patches to code.opencv.org
  • Fast normal finder
• Release April 2
• Khronos … lots of nitty gritty detials. This is going to take awhile …
• Non-profit
  • This is on, many efforts and considerations
  • We will make this happen within the month.
• Next week Victor in the US
• New user page for OpenCV … should be up within a few weeks.
  • User site will be: OpenCV.org vs
  • developer code.opencv.org
  • Look into moin moin for the wiki
• 3D needs. Filters, normals, voxel filters
• Eigen
  • Good community. Large library
  • 2 branches for optical flow with and without
  • OpenCV has small matrices 4×4, 3×3 MatEx structure
    • Add quaternions to MatEx
  • Nice geometry features in Eigen
  • Yet another dependency
    • It had issues with misaligned data, goes to just C code
      • Might be handled in new version
  • Need better support for simple translation
  • Add a few routines to MatEx, Gary will put up a page on code.opencv.org to collect what is needed
• Need general Levenberg-Marquard
  • Simple optimization is needed, some of this is embedded inside machine learning
  • Expose, normalize and justify the RANSAC that is there
  • lmdir lmdiff
  • sparse bundle adjustment
  • libmv will give a bunch of these functionalities
• Connected components — findContours repeated calls of floodfills
  • Kurt has CC for speckle components
• VO, Maria is on this.
  • Not getting quite the same accuracy as the paper yet (missing details in paper)
  • Efficiency looks good, 30-50ms/frame on CPU … could be GPU’d later
• Transparent objects
  • Ilya has done good work. Gary will collect data from new sets and perhaps from new contacts from Proctor & Gamble
  • We will combine several techniques to increase reliability

Ilya

RGB-D odometry [Maria]
Added the RGB-D implementation and the sample to the OpenCV contrib module.

Implemented pure rotation and pure translation cases. Added ability to initialize transformation between frames.

Tested the implementation on the Kurt’s dataset. There is only camera translation along a textured plane there. We don’t have ground truth, but visually the method works well on the data. To compare RGB-D odometry results with given kinfu results we plot trajectories (see trajectories.gif). It seems RGB-D odometry is more accurate on the data, especially if pure translation is estimated. RGB-D odometry for rigid body motion accumulates a bigger error during ~350 frames than for pure translation, but it looks better than kinfu icp results.
Transparent objects [Ilya]
Wrapped SimpleBlobDetector for Python and added a full Python wrapper of findCirclesGrid.

Adapted our old dataset of 5 transparent objects for evaluation of glass segmentation. Dataset images are segmented automatically into glass, background as behind the glass, and other background by using fiducial markers. So ground truth mask is computed. The segmentation is computed using the OpenCV Python interface.

Implemented infrastructure (in Python) to evaluate glass segmentation. It runs the algorithm on all images and computes precision and recall of the glass segmentation. Experiments are on the way.
OpenCV infrastructure and issues [Alexander Shishkov]
OpenCV infrastructure:

- fixed a problem with concurrent work of Android builders on Buildbot (#1274)

- investigated a problem with MinGWLogObserver, tried to fix it, but I still don’t understand the reason of such behaviour (#1303)

- installed newest MinGW/MSYS tools to the new Buildbot server, implemented alpha version of the builder for the FFMPEG wrapper (#1266)

- conversation with the Askbot main developer and different customers of OSQA and Askbot, investigation of Askbot administration. I tend to choose Askbot as framework for OpenCV Q&A forum now.
OpenCV issues:

- added support of XtionPRO/XtionPRO Live, updated a sample and documentation (#1657)

- replaces all references to code.ros.org, opencv.willowgarage.com/documentation, opencv.jp to the actual resources (code.opencv.org, opencv.itseez.com, etc..); fixed some bugs in documentation; updated cheatsheet.pdf (#1672)

- fixed some compilation and documentation errors (#1554)
OpenCV tests and bug fixing [Alexander Reshetnikov]
Added the first version of the new ffmpeg wrapper into OpenCV (for versions from 0.7 to 0.10). Now the used wrapper is chosen depending on ffmpeg version.

Current implementation doesn’t work correctly with C interface (highgui tests with video i/o terminated unsuccessfully with hardware/os exception). So now the main activity is fixing the bugs in C interface of the new wrapper and checking it on existing highgui tests.
Also decided that the highgui module needs an another positioning test where we create specific video (every frame of this is binary image that has byte code of frame number).

This test will be independent of compression algorithms used in different containers and codecs.
Other
March 8 was a national holiday in Russia.

Vadim

Anatoly

Action Items

Gary

• Submit ECCV workshop
• Put up a wiki page on developer site for all the rotation MatEx operators that we need
  • Eigen support
• Put in ticket for connected components generalized from Kurt’s speckle filter

Vincent

• Call Ethan to get latest opencv in ROS
• Help with moving libmv along … use GSoC resources to help??

Alexander

Vadim

• Look into moin moin for wiki
• Look at pulling connected components from speckle filters (Gary will ticket)

Victor

Kurt

Ethan

From Last Time

Gary

• (./) Finish GSoC application http://code.opencv.org/projects/gsoc2012/wiki/Gsoc2012
• Submit ECCV workshop
• (./) Contact James about limitations in 2D barcode
  • Can be lifted, probably not before next release
• (./) Ask Kurt about visual odometry 2D+3D

Vincent

• ROS integration with latest possible OpenCV trunk
• Describe OpenNI for ASUS prime sense

Alexander

Vadim

• Talk to Anatoly about stand along kinfu

Victor

Kurt

• (./)Talk to Maria on visual optical flow with rgbd
  • Great progress, dense optical flow is fast and seems to work quite well
• Near term goal is to integrate code into OpenCV
  • In contrib now
• Even more robust motion estimation with pure translation assumption
• Robust detection of visual tracking failure

Ethan

• Get OpenCV for Asus prime sense

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Google Summer of Code. Priorities, see http://code.opencv.org/projects/gsoc2012/wiki/Gsoc2012
  • Select projects, spread the word, talk to students
• Ethan stuff,
  • fast structured normals
  • OpenCV for Asus prime sense
• Eigen
• Rotation convenience
• Khronos updates
• OpenCV.org progress
• Website … how to put up new initiatives such as seemless rotation and translation forms in code.opencv.org
  • Rotations convenience functions
    • Quaternians, Rodriges, R,t and just some common functions we face (segmentations, histograms, morphology)
• libmv

Minutes

• GSoC Mentors start talking with students
• RGBD modules dense and another module sparse
  • Convenience function for rotating colored pt cloud and then dealing with it as an image (getting a “birds eye view”)
  • Modified Kinfu should come from IPI
  • Sparse points such as SFM
• Maria Kurt VO
• With ASUS Action Pro (“kinect camera”) — should work fine now.
  • Now you can open and run both
• Eigen
  • Conversion functions back and forth
  • Need sample, tutorial, doc
• Khronos
  • Nice to get a standard, might take 2 years … OpenCV will comply with it when the day comes
• OpenCV.org
  • Local guy Ramji who is helping Gary with filing the paperwork
  • Making a case to a potential big sponsor
• libmv … Will contact back in April … so, not this release.
• On line classes for vision vision.class.org by Malik (March 2012)
• Website graphics guy and website maintainer
  • Use common framework, maybe drupal
• Code freeze next week, Beta release of 2.4 next week
• Latest OpenCV is now in ROS, no problems integrating … thanks!
• Gary is traveling on IPI business all next week, so no meeting next week, see you in 2.

Ilya Lysenkov

Transparent objects [Ilya]
Added computation of full recall-precision curves to evaluate glass segmentation algorithms.

Experimented with the pure voting algorithm and Geodesic Active Contours. Curves showing advantage of Geodesic Active Contours are attached.

Investigated and fixed two bugs in computation of invalid regions and aggregation of SVM predictions.

Removed hard-coded parameters, used the Python tempfile module to generate temporary files, removed debug info writing.

Combining of the RGB-based and depth-based glass segmentation algorithms is in progress.
OpenCV development and tests [Alexander Reshetnikov]
New wrapper supporting ffmpeg versions from 0.7.x was refactored (changed implementation for opening, closing, setProperty and getProperty methods, grabFrame and retrieveFrame functions).

Also it was checked on frame count test for versions 0.7.11, 0.8.10, 0.9.1 and 0.10. It has given following results (actual frame count in every video is 125):

Started the work with new positioning test (see  checking new wrapper on positioning and writing tests.

Compilation errors under Ubuntu with ffmpeg 0.7.x were occurred with new wrapper. Some problems have been already fixed. Open issues with it will be closed in the nearest time.
OpenCV infrastructure [Alexander Shishkov]
Buildbot:

- updated one of the test servers to the Ubuntu 11.10 (#1068)

- added time-stamp to the Report page (#1329)

- fixed problems with highgui tests on GUI (#1300)

- added new builder for updating opencv_ffmpeg(32/64) dlls after changes in FFMPEG wrapper (#1266)

- added new builder for test highgui module with different versions of FFMPEG library 0.10, 0.5.1, 0.7.3, 0.7.8 (#1272)
Other:

- preparation to the OpenCV hackaton (instructions, issues selection, discussions)
KinFu and RGB-D odometry [Maria]
Implemented initial version of approximate synchronization of Kinect image and depth. Synchronized RGBD data is necessary e.g. for RGBD odometry. The synchronization is supported in both single threaded and multi-threaded versions. Single-threaded version may be used when there isn’t TBB and frames are grabbed from video (i.e. frames aren’t skipped). But reading/writing .oni video files aren’t supported now. Multi-threaded version based on TBB can be used to synchronize data grabbed from the camera online.

Experiments with the initial implementation is in progress. We are trying to minimize the number of skipped online-synchronized “image-depth” pairs by balancing work between threads and playing with synchronizer parameters.

Vadim

1. A lot of changes made locally on my machine during winter have been finally committed to SVN repository. It includes quite substantial refactoring of features2d: introduction of Feature2D class that combines feature detector and descriptor extractor, using the super-abstract class Algorithm for Feature2D (Algorithm will also be used as a base for StereoMatcher, BackgroundSubtractor, OpticalFlow etc.), and many other simplifcations.
1.1. SIFT & SURF are moved to a separate nonfree module, so that it’s possible to ship OpenCV binaries free of any patented algorithms (i.e. in future it’s planned to move more stuff to nonfree).
1.2. Brute-force matcher is now threaded using TBB.
1.3. Some obsolete stuff, like Calonder, Ferns etc., as well as C interface of the module, has been moved from feature2d to legacy module.
2. The new module “photo” has been added. Right now it just contains inpainting algorithm; it’s planned to add more computational photography stuff there in future (maybe even before 2.4 release).
3. Enhanced variant LogPolar transform (using so-called Blind Spot Model), contributed by Fabio Solari and Manuela Chessa, has been put to contrib module.

Anatoly

Action Items

Gary

• Talk to Kurt et al for what we want pragmatically for rgbd

Vincent

• Send emails to find mentors

Alexander

Ilya

Vadim

• Create Eigen tutorial/doc, sample for using Eigen

Victor

Kurt

• Get back to Maria on VO

Ethan

From Last Time

Gary

• Submit ECCV workshop
• Put up a wiki page on developer site for all the rotation MatEx operators that we need
  • Eigen support
• (./) Put in ticket for connected components generalized from Kurt’s speckle filter

Vincent

• Call Ethan to get latest opencv in ROS
• Help with moving libmv along … use GSoC resources to help??

Alexander

Ilya

Vadim

• Look into moin moin for wiki
• Look at pulling connected components from speckle filters (Gary will ticket)

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• GSOC
• Transparency
• Release

Minutes

• Demo when Vincent gets back: Maria and Ilya working on.
• OpenCV resources
• OpenCV.org coming soon hopefully.
  • Alexander on another project. Eugene replacing.
  • Funding OpenCV
    • Sponsor areas
      • Computational photography, SFM,
      • Architectures (ARM … Xilinx)
    • Summer schools
    • Travel support
    • Workshops
    • Let’s say, by the end of May it’s up and running.
• GSoC … mentors need to sign up now for OpenCV
• Release
  • Beta packages out this week!
  • Final release by end of April
  • ~300 bugs fixed
  • Stabilization was a big issue on this release. Thanks to NVidia Tegra team especially!
    • Tegra team is going to go to quarterly releases, then perhaps monthly
• Workshop
  • Set up website
  • End in hackathon
  • Develop OpenCV Schwag
    **

Vadim

Summary

2.4 beta has been delayed by 1 more week to make our buildbot “green”, i.e. eliminate the remaining outstanding problems. The goal has been mostly achieved (>15 failures have been fixed); now the packages are being prepared and will be uploaded to SF this week.

~10 more tickets have been fixed last week.

Refactored API of the recently rewritten Expectation Maximization algorithm; fixed several bugs and provided Python interface for it.

Multiple bug fixes in ffmpeg wrapper (also very problematic part of OpenCV); now some of the related tests pass, but some still fail; we will look at it after 2.4 release.

Documentation hackathon has been just started; we fix documentation-related issues in the tracker and add the missing documentation.

Details:
ffmpeg stabilization [Alexander Reshetnikov; Vadim]
Accomplishments:

1) fixed some problems with VideoCapture in new ffmpeg wrapper: (crashes on avi files, incorrect frame reading);

2) closed 4 tickets (# # 1708, 1755, 1743, 1287), # # 1112, 1736 are in progress;

3) minor changes in highgui structure

4) fixed crashes in video writer in the case of odd-size videos; fixed crashes in VideoCapture in non-indexed containers, e.g. .mpg files; fixed finalization of video streams in video writer; fixed some of ffmpeg tests to avoid obviously invalid cases; disabled too verbose logging of ffmpeg codecs;
Plans:

1) further bug fixing in highgui;

2) complete performance tests for video i/o;

3) participate in documentation Hackathon on this week
Intrastructure, eliminating test failures [Alexander Shishkov]

- fixed 9 OpenCV tests

- installed plugin for hiding emails by default on OpenCV Redmine, changed this setting for current users

- added automatic Redmine reboot with insufficient memory (memory leak in Redmine)

- fixed environment for test_ffmpeg and coverage builders on Buildbot

- modified WinSuperPack builder (fixed some bugs, added additional files to pack)

- modified online documentation theme (removed tool-tip below QuickSearch)

- fixed ##227,1752
Other:

- bought BNC-F connectors

- delivered certificate about “Argus Procam” software registration

- agile and OpenCV+Tegra meeting
Expectation Maximization improvements [Maria; Vadim]
- Found and fixed a few bugs in kmeans.

- Refactored interface of cv::EM. It has a python wrapper now.

- Started to solve the tickets on EM (#1407, #1581).
Plans

More tests on EM, doc on EM.
Misc
Ilya was on vacation last week.

Action Items

Gary

Vincent

Alexander

Ilya

Vadim

Victor

Kurt

Ethan

From Last Time

Gary

• Talk to Kurt et al for what we want pragmatically for rgbd

Vincent

• Send emails to find mentors

Alexander

Ilya

Vadim

• Create Eigen tutorial/doc, sample for using Eigen

Victor

Kurt

• Get back to Maria on VO

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Mentors, choose students here prior to April 20th!.
  • We need a repository where we can see this code
• Linemod
  • Go over details
• Release
• New website
• OpenCV.org, OpenCV non-profit foundation
  • All documents are filed, waiting for Employer Identification Number and ficticious name certificate to be returned.
• Workshop status?

Minutes

• Choose your students.
• We have a separate project on code.opencv.org for GSoC projects. We can move things in from there.
  • Python samples should go right in
  • Docs and tutorials go right in
• Linemod
  • Post issue in bug tracker
  • Better model capture maybe using Maria’s
    • Knowing if the camera is super useful in any case (in contrib)
      • 2 images; 2 depth maps/ 2 masks (regions to be mapped) odometry
• Code.opencv.org was attacked
  • Got access to server
  • This attack was from China. Stop it.
  • Alexander cleaned it up
• Beta release is out. Final release by end of month
  • Change log https://github.com/opencv/opencv/wiki/ChangeLog
  • On ROS side, Fuerty. Some build problems with the farm,
    • SURF and SIFT was moved to “non-free” and this messed up the ROS build. But OK
  • but Vincent will sync final release to this
• New website for users
  • Design is ready
  • Web programmer was hired away, hiring another web company. Probably 2 weeks.
  • Contests etc … need guy on this. Probably case by case
  • Itseez hiring web programmer, can use this guy part time initially
  • Web framework will be a well known engines (maybe Drupal)
  • Redmine for code.opencv.org
• Transparent objects
  • Experiments show we can recognize in clutter
    • Robot can grasp one by one (but robot has to be aware of where other objects are)
    • Does this make sense for June demo at Willow? Most of the ROS stuff is done for this
      • Probably go for it
  • Vincent coming end of May
• Last week was documentation hackathon
• OpenCV docs have fewer broken links etc

Vadim

OpenCV 2.4 beta is out [OpenCV team]

OpenCV 2.4 beta is finally out. Here is the announce:

http://tech.groups.yahoo.com/group/OpenCV/message/87261 and

here is the changelog: https://github.com/opencv/opencv/wiki/ChangeLog
OpenCV Documentation hackathon is over [Ilya and others]

Last week we had documentation hackathon. Since beta 2.4 has been delayed by one week, and so we had some overlap in schedule, it was not as effective as the bug fixing hackaton. Still, we closed 24 documentation-related tickets in our tracker, 19 of which was closed by Ilya! The documentation will further be improved until and after the final 2.4.
EM improvements [Maria, Ilya, Vadim]

Maria, with help from from Ilya and Vadim, further improved EM implementation. The new implementation has automatically generated Python interface and much better interface. Several EM-related tickets have been closed as a result of this refactoring: #964, #1340, #1351, #1361 (according to our bug tracker, EM was quite commonly used and yet quite problematic part of OpenCV; hopefully, most issues are now resolved).
Improving ffmpeg wrapper [Alexander Reshetnikov]

Implemented 2 new highgui performance tests for video i/o.

Closed the tickets: #1771, #1790, #203, #1736.
Other bug fixes, improvements [Maria, Vadim]

Other mll and features2d bugs/improvements:

Improved MLData to read files with different space count between columns.

Fixed the ERTrees name in saving/loading;

Fixed #1798, #1795, #1805.

Fixed crash in linemod detector, reported by S. Hinterstoisser.

Extended morphological operations to handle more than 4-channel images, 16S and 64F formats, added test that checks supported formats in filtering operations.
Infrastructure [Alexander Shishkov]

Buildbot:

Updated Windows SuperPack builder (added the skipped include files, fixed incorrect include files for the stitching,…).

Transferred Android builders to MacOS.

Added the OpenCV testing with FFMPEG 0.10.2.

Fixed the problem with svnsync on Buildbot.
OpenCV 2.4 beta packages for Windows/Unix are prepared and uploaded to the SourceForge.
Eliminated the consequences of the attack on the code.opencv.org site (checked for rootkits, removed enemy processes, added the using SSH keys, added IP list for SSH).

looks like we are getting popular :)
Added DNS docs.opencv.org (link to opencv.itseez.com).
Reviewing GSoC proposals [Vadim, Anatoly]

Finished reviewing all the submitted proposals. This year we’ve got a lot of proposals, so while the majority of proposals can be easily rejected, there are enough good ones to choose from.

Action Items

Gary

• (./) Start a opencv.org “wish list”: What we can do with what funding
• (./) Get P&G objects for transparent test

Vincent

• See about converting Maria’s tracking to a generic model capture of camera pose. Talk to here
• Sync final release with ROS

Alexander

Ilya

Vadim

• Write a hello world application for GSoC 2012

Victor

• Talk to NVidia about workshop

Kurt

Ethan

From Last Time

Gary

Vincent

Alexander

Ilya

Vadim

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Name reserved for OpenCV.org
• GSoC … Mentors discuss with plans interns
  • Focus for object rec planning
  • Distribution tracker
• Release
• New website
• Workshop
  • ECCV
• Post 2.4 plans
  • Transparent objects

Minutes

New paper by OpenCV dudes to be presented at RSS:

“Recognition and Pose Estimation of Rigid Transparent Objects with a Kinect Sensor” by Ilya Lysenkov, Victor Eruhimov and _Gary Bradski, RSS 2012.

• OpenCV.org is “taken” probably by us. Open Source Computer Vision Library is reserved. Will be filing this week.
• GSoC
  • Training classifiers
    • Silverware
      • Tracking from last year.
      • There are classifiers there: Multiple instance learning: mil-boost
      • RSS workshop rgbd sensors, plane finding
      • Shape context into the cascade classifier …
      • LARK?
• Release. Beta out now
  • 2.4 final will be out this coming weekend
• New website for users (existing new website for developers is /|here)
  

  
  
  Update on the opencv.org site. We have started working with the vendor last week, we expect the first version of the site by the end of the next week and the final version one week later. Given potential delays and time for content creation, it is reasonable to expect site rollout by the end of May.
  
  

  • Hot or not in vision http://www.cvpr2011.org/statistics
    • Blob segmentation
  • Or, create way for users to drive what they want
    • http://brainstorm.ubuntu.com/
• Instead of yahoo groups, switch to askbot so that user questions are answered quickly
  • Then mailing list would be more about announcements: “Hey, I released this”.
• Tutorial at CVPR …
  • Need to submit also to ECCV
  • $2K award for best open source code from CVPR submitted to OpenCV.
  • RSS July 9th … need travel funds
• Vincent will be local May 21st, Ilya will probably come too
• Post 2.4 plans
  • RGBD module
    • Linemod
    • structured arrays
  • Split OpenCV into many smaller modules by topic
    • Output can be single large library, or small custom fit libraries
      • Keep dependencies to a minimum to allow the user to minimize space if they want
      • Naive users or those who don’t care can produce single dynamic or static library
  • Computational photography will be a big topic
    • Samples should be mobile demos in iOS and Android
  • Video stabilization, tracking, effects, color or white balance, augmentation
  • Mobile application: Food detection … this is hard
  • Hot or not in vision http://www.cvpr2011.org/statistics
    • Blob segmentation
  • Or, create way for users to drive what they want
    • http://brainstorm.ubuntu.com/

Anatoly

Accomplishments & statuses:



1)      Video Stabilization.

·         Fixed trim ratio estimation bug for the case of homographies motion model.

·         Ported some parts to GPU (overall speed-up ~3x)

o   Added GPU version of wobble suppressor (14x faster than CPU version). Replaced DECOMP_SVD in global motion estimation methods with DECOMP_NORMAL|DECOMP_LU (it works about 2x faster).

o   Added GPU version PyrLK based global motion estimator (overall speedup of global motion estimation is ~8x, that's about 22 seconds for 500 frames).



•     Started working on local outlier rejection method described in the YouTube stabilization method paper (going to check whether it can improve motion estimation quality).


3)      OpenCV GPU.

·         Integrated VideoWriter_GPU and VideoReader_GPU to OpenCV trunk, results see below. Added tests/docs for GPU decoding/encoding.

·         Tested compilation with CUDA 4.2 under Windows and Linux.

o   Issue: One BruteForceMatcher_GPU test fails for 4.2 (match with collection with mask with L1 distance). Need to investigate.



·         Tested performance. The numbers below include NVidia Encoder/Decoder library performance together with our code.


Encoding:
Windows only:            1920x1080                          4096x4096
CPU [XVID]      21.3 ms/frame [FPS :  46.9]      175.4 ms/frame [FPS :   5.7]
GPU [H264]      1.8  ms/frame [FPS : 546.3]        7.4 ms/frame [FPS : 135.5]
For 1920x1080 default parameters was used, for 4096x4096 bitrate was increased.

Decoding:
Windows:       1920x1080 [MPEG4]                 1920x1088 [H264]
CPU       4.6 ms/frame [FPS : 218.4]      2.8 ms/frame [FPS : 359.9]
GPU       2.7 ms/frame [FPS : 350.6]      4.5 ms/frame [FPS : 224.4]

Linux:         1920x1080 [MPEG4]                 1920x1088 [H264]
CPU      10.9 ms/frame [FPS :   91.8]            <doesn't work>
GPU       1.0 ms/frame [FPS : 1002.0]        7 ms/frame [FPS : 142]

VideoReader_GPU can't open 4096x4096 video file. (it seems NVidia doesn’t support).




Plans:

·         PCL: Continue Skeleton Detection optimization with/without GPU.

·         VS: Continue working on local outlier rejection in the global motion estimation step. Try using L1 minimization (using LP simplex solver) instead of L2 when estimating global motion (as described in the youtube stabilization method paper).

·         OpenCV: Fix BruteForceMatcher_GPU. Investigate issues #1831,  #1823.

Ilya

Transparent objects [Ilya]
1. Tested our pose estimation algorithm in transparent clutter to get a proof of concept whether a robot can clean out a table with transparent objects.

So several objects were placed on a table, our algorithm returned pose of the recognized object with the best confidence, this object was removed from the table and this procedure was repeated. Results are shown in the animated GIF:

2. Optimized the pose estimation algorithm. Now silhouettes of a model are computed using dot products with normals instead of morphology. Re-scanned objects to get models with more accurate normals (thanks to Maria for granting access to her computer with kinfu). Experiments show that the optimized algorithm is almost 2x faster than the old algorithm but accuracy is worse. It means we can combine two algorithms together by finding the single pose with the optimized algorithm and then refinining it with the old algorithm to get fast pose estimation with good accuracy. Results are shown in the table:

-----------------------------------------------------------------------------------------------------
|        |  Mean translation error (cm)  |   Mean pose estimation time (s)    |         Speed-up    |
| Object |-------------------------------|------------------------------------|---------------------|
|        |  old   | optimized | combined |   old   |  optimized  |  combined  |  optimized |combined|
|---------------------------------------------------------------------------------------------------|
|bank    |  0.41  |   0.68    |   0.40   |   31.1  |    20.4     |    20.4    |     1.5    |    1.5 |
|bottle  |  1.41  |   1.33    |   1.37   |   22.6  |    10.7     |    12.1    |     2.1    |    1.9 |
|bucket  |  0.34  |   0.71    |   0.45   |   55.7  |    26.4     |    26.3    |     2.1    |    2.1 |
|glass   |  0.48  |   0.57    |   0.50   |   19.7  |    10.4     |    10.7    |     1.9    |    1.8 |
|---------------------------------------------------------------------------------------------------|
|mean    |  0.66  |   0.82    |   0.68   |   32.3  |    17.0     |    17.4    |     1.9    |    1.8 |
-----------------------------------------------------------------------------------------------------

So the final algorithm has the same accuracy as the old one but it is 1.8x faster.

Other:
Our paper "Recognition and Pose Estimation of Rigid Transparent Objects with a Kinect Sensor" by Ilya Lysenkov, Victor Eruhimov and Gary Bradski has been accepted at Robotics: Science and Systems Conference (RSS) 2012.


RGBD Odometry [Maria]:

Started to work on the model capture demo. Before integrating the RGBD Odometry I was going to run the existing capture pipeline on a dot pattern or an ORB template. It is in progress now because there are problems with setting up the working environment and import/compilation/build of the pipeline.

Vadim

OpenCV stabilization & bug fixing [Alexander Reshetnikov, Vadim]
Closed 15 tickets, including 5 related to video capturing.
Further EM improvements [Maria Dimashova]
- added doc on cv::EM, moved doc on CvEM to legacy/doc (#8096);

- resolved failure of EM accuracy test on ubuntu x86 (32-bit);

- resolved failure of EM save-load test.
Background segmentation improvements [Vadim]
Cleaned up MOG2 code; Threaded it using TBB.
OpenCV infrastructure [Alexander Shishkov]
OpenCV Wiki from opencv.willowgarage.com has been converted to the redmine format (temporarily available here: http://redmine.alekcac.webfactional.com/projects/wiki/wiki/index)

Extended buildbot with new tests (nonfree, stitching & photo modules), all performance tests now run in “sanity check” mode to do some more regression testing of the library.

Moved windows superpack builder to a dedicated workstation and brought the build time from 23 hours down to 3 hours! (big thanks to Andrey Kamaev from Tegra team)
Proposal for Vision standard API [Victor, Vadim]
updated our proposal on vision API standard (aka OpenVL) for Khronos F2F in Dublin. The proposal is based on the collaborative work with experts from ARM and Samsung. Victor has presented this proposal on Tuesday.

Action Items

Gary

• Talk to Cat about workshop support for OpenCV tutorial
• Apply for ECCV tutorial
• Kick people to talk to GSoC interns now. Computational photography is high priority
• (./) Make sure Alexander Shishkov takes into account something like http://brainstorm.ubuntu.com/ in opencv site

Vincent

Alexander

Ilya

Vadim

• When 2.4 final is out, tell Vincent

Victor

• Get plan for CVPR tutorial in place

Kurt

Ethan

From Last Time

Gary

• (./) Start a opencv.org “wish list”: What we can do with what funding
• (./) Get P&G objects for transparent test

Vincent

• See about converting Maria’s tracking to a generic model capture of camera pose. Talk to here
• Sync final release with ROS

Alexander

Ilya

Vadim

• Write a hello world application for GSoC 2012

Victor

• Talk to NVidia about workshop

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Release
• New website
• Workshop
  • ECCV
• Post 2.4 plans
  • Transparent objects

Minutes

• Release 2.4 is out!!! 2.4 Packages log]], [[http://sourceforge.net/projects/opencvlibrary/files
  • On the ROS side it is released as well (ROS Fuerte)!
• New website
  • Designer is on vacation, mock site is up
• CVPR workshop (June . Now that 2.4 is out, we can start preparing for this
  • iOS is biggest risk because we don’t have someone on this. Vadim is on this.
  • ECCV workshop proposal is in.
• Post 2.4
  • Tegra team has grown: Want to release each month
    • They can help from infrastructure point of view until (hopefully if) OpenCV foundation is formed.
  • Short term: Mobile focus to CVPR
  • Computational photography
    • Collage, video summary
  • Many more modules
    • Library and application side
    • OpenCV apps site. Itseez seeds
      • Might turn into an apps store for OpenCV.org
      • Standard directory structure with email
      • Image stitching started as an app then turned into a module
    • Make a test case. Computational photography is one.
• GSoC
  • SfM … libmv 3rd guy willing to work for fun from Hartley’s group

Vadim

OpenCV 2.4.0 is out [All the team + GPU + Tegra guys]
Finally, we have shipped it! Thanks to everybody who participated!

The changelog is here:

https://github.com/opencv/opencv/wiki/ChangeLog

The packages are here:

http://sourceforge.net/projects/opencvlibrary/files
The announcement was sent to the mailing lists.
Bug Fixing [Alexander Shishkov, Alexander Reshetnikov, Vadim, GPU&Tegra teams]
~20 more tickets have been closed during the last week. A few more serious bugs in ffmpeg wrappers have been fixed and now it [finally?] works quite well.
Misc.

Holidays: as Kurt once said – lot’s of national holidays is good, it helps to unite the nation. These days we have a bunch of such opportunities. This Monday and Tuesday, as well as the next Monday and Tuesday are holidays, while the last and this Saturday are working days.

Anatoly

Accomplishments & statuses:
1)      Video Stabilization.
·         Implemented local outlier rejection. Can’t say it increased motion estimation stability a lot.
·         Implemented linear programming based motion estimation (it minimizes L1 norm, which is less sensitive to outliers). It works more stable (significantly) than RANSAC version, but slower.
·         Replaced CV_RANSAC with CV_LMEDS in findHomography. Visually, it increased stability a bit on some videos.
Issues:
·         OpenCV trunk can’t be compiled using gcc 4.6.3 because of internal error in gcc (bug #1860). The bug was revealed during compiling videostab module.
3)      OpenCV GPU.
·         Refactored gpu::BruteForceMather (made it similar to new cv::BFMatcher).
·         Fixed NVidia/NCV.Visualization test (drawRects in CascadeClassifier_GPU).
·         Fixes bugs #1831, #1823, #1759.  Investigated and fixed bug in Face Detection that was committed by Tegra team when they were fixing warnings everywhere.
4)      OpenCV Demo Pack preparation for GTC
·         Fixed several bugs (like ‘window size can be set to zero at real time’)
·         Updated optical flow sample, split it to 2 samples – dense and sparse, added frame interpolation demonstration to dense sample.
·         Stereo demo was split into two:
o   BM match only (CPU vs. GPU), which runs on high resolutions.
o   BM vs. BP vs. CSBP which runs on lower resolutions due to memory limitations.
Plans:
·         Prepare datasets and stitching demo. Possibly rework feature detection demo.
Plans:
·         OpenCV: Finish with GTC demo pack (update feature detection/matching samples ).
Other:
Alexey will be on vacations 2-31 of May. After he will joint Tegra team. Instead, Marina will come to GPU project from Tegra team.

Action Items

Gary

• Mail ideas to Anatoly Baksheev
• Mail Ilya pictures and interact. Thursday.
• Contact Carol about GSoC who may be trying to be part time.

Vincent

Alexander

Ilya

Vadim

• Find out status of new user website

Victor

Kurt

Ethan

From Last Time

Gary

• Talk to Cat about workshop support for OpenCV tutorial Waiting on EIN
• (./) Apply for ECCV tutorial
• (./) Kick people to talk to GSoC interns now. Computational photography is high priority
• (./) Make sure Alexander Shishkov takes into account something like http://brainstorm.ubuntu.com/ in opencv site

Vincent

Alexander

Ilya

Vadim

• When 2.4 final is out, tell Vincent

Victor

• Get plan for CVPR tutorial in place

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• CVPR tutorial (May 15th Early registration deadline
• Travel
• Foundation: EIN number received
• Website
  • Contribution
• GSoC: May 21 official start, can start earlier.
  • One student on agglomerative mean shift segmentation
• New stuff
  • Inverted index
    • Need a template for this

Minutes

• Use fieldhouse.com for schwag (used by ROS)
  • CVPR committee about shipping stuff there
  • May 15th early registration deadline
• OpenCV.org, now have EIN number
  • Click for donations needs to be set up. Use Paypal probably upgraded to business account
  • Use fieldhouse for schwag
  • Need to set up security for new site
• Khronos
  • Ponderous discussions of API …
• CVPR Tutorial. June 17th. May 15 Early registration deadline
  • We want to offer lunch etc, but have to wait until OpenCV has an account and gets funded.
  • Put up registration and try to get it going. Deadline for prior registration needed.
  • How much time will fieldhouse take to make schwag.
    • Probably a week or 2
  • Deadline for our workshop registration will be June 3rd
• Travels
  • 2 people traveling to CVPR. Victor and Vadim for the tutorial.
  • Ilya and Maria over
  • Paper accepted for RSS (Ilya, Victor, Gary). Ilya going to RSS to present. Early in July 9-13
• GSoC Start your engines interns May 21
  • SfM One from Hartly group.
    • b2solver released by Google
  • Python bindings (has 2nd internship)
    • If there are samples people want, name them
  • Super resolution for email exchange
    • state of the art dense optical flow
    • super resolution for computational photography module
• Website
• Computer vision mailing list. We should post there.
  • imageworld, http://www.vislist.com/

Ilya

Transparent objects [Ilya]
Solved the problem of false positive detections with a very small scale. Tried to make scale normalization of the cost function theoretically grounded by using a Bayesian formulation but it didn't work well in practice. Ended up with simple thresholding on the minimum scale.
Created a new dataset with transparent clutter and ground truth poses to evaluate our algorithm in clutter. The previous optimization with normals doesn't work well on this dataset so the algorithm was optimized by other means (for example, suppression of poses after alignment to a table plane, removing iterative refinement of the closed-form solution, sampling a test silhouette). Results on this dataset (the success rate is the proportion of cases when the translation error is less than 2cm):

-------------------------------------------------------------------------
Object         Success rate    Mean translation    Mean pose estimation
error (cm)              time (s)
-------------------------------------------------------------------------
bank               0.86              0.4                   17.5
bottle             0.98              1.1                   13.1
bucket             1.00              0.2                   18.0
glass              0.88              0.6                    8.7
sour_cream         0.95              0.4                   13.6
-------------------------------------------------------------------------
mean               0.93              0.5                   14.2
-------------------------------------------------------------------------

So results are slightly worse than pose estimation in clutter-free environment but still they are good enough for grasping. The optimized algorithm works the same time as on the dataset without clutter.

Also accuracy of initial poses computed by geometric hashing was evaluated alone. The accuracy is measured as the minimum Chamfer Distance between projected silhouettes and the ground truth silhouette. Results (the success rate is the proportion of the cases when this Chamfer Distance is less than 5 pixels).

------------------------------------------------------------------
Object         Success rate     Mean Chamfer     Mean number of
Distance (px)     initial poses
------------------------------------------------------------------
bank               1.00             0.7               27.0
bottle             0.98             1.1               41.7
bucket             1.00             1.8               33.8
glass              0.66             1.5               20.2
sour_cream         1.00             0.8               32.0
------------------------------------------------------------------
mean               0.93             1.2               30.9
------------------------------------------------------------------

So geometric hashing produces the accurate initial pose for our algorithm in the most of cases.

Other:
I got the Australian visa to attend RSS 2012.


RGBD odometry [Maria]
Built the ecto-based capture pipeline. It required some effort: to install ubuntu 10.04 (my installed 10.10 isn't supported by ros fuerte), to install ros fuerte, to fix several build problems of the pipeline - thanks a lot to Vincent for this.
Played with the pipeline scripts which are used for model capture based on orb template.
Experimented a little with the pipleline sample of rgbd odometry. The algorithm diverges in several cases:
1) there are moving objects in the scene or changing light (the algorithm supposes a statical scene)
2) there is a motion blur
3) rgb and depth are not synchronized (the ecto openni wrapper doesn't support such synchronization, approximated by time stamps).
Worked on improving the rgbd odometry accuracy:
1) Median filtering of rgb and depth increases slightly the accuracy on the TUM dataset.
2) Implemented a cpu version of icp from kinfu (only camera pose estimation, without a scene volume). Initial implementation uses a random subset of close points and works ~0.15s on one core of core i5 (I think it can be optimezed up to rgbd odometry time ~0.08s). The implemented icp gave the relative median pose error on the TUM dataset comparable with the rgbd method, but its trajectory was further from ground truth than rgbd's and kinfu's ones. Tried to combine rgbd and icp (to minimize not only photo consistency error but spatial error too) but again the way that improves the merged approach quality on the TUM dataset wasn't found. Will try the icp for model capturing (maybe it will be useful).
3) Started to study BA from Vincent.


Other:
We have 2 national holidays these weeks.

Vadim

Closing tickets [Vadim Pisarevsky]

6 tickets have been closed, mostly orphan ones.
opencv.org [Alexander Shishkov; our (contracted) web masters]

The first draft of opencv.org is ready (the link was sent by a separate e-mail; not to be disclosed yet).

Thanks everybody who provided feedback on the design and content.
OpenCV roadmap meeting [OpenCV, GPU & Tegra teams]

There was the meeting at Itseez regarding the possible and feasible release schedule for this year. I can not find any summary sent, but here is a short summary:
OpenCV 2.5 is expected to be out around Nov 1st.

Before that there will be at least two 2.4.x releases, may be more.

Also, we plan to make 2 hackathons – in the end of June and in September.

After some friendly conversation we seem to agreed to migrate to git in order to simplify the simultaneous work of 3+ teams working on OpenCV. One of the primary reasons to choose git vs mercurial – github and its “pull request” feature, using which we can react on most feature requests and bug reports with known solutions in a timely and friendly way – “please, do it yourself and submit a pull request”. It’s definitely better than leaving such tickets without attentions for years.

Action Items

Gary

• Call Silicon Valley bank
• Send Ilya transparent items photos

Vincent

• Contact Bianca about fieldhouse and producing runs of schwag
• Push for travel access

Alexander

Ilya

Vadim

Victor

• Get Vincent’s comments in to website designer

Kurt

Ethan

From Last Time

Gary

• (?) Mail ideas to Anatoly Baksheev
• Mail Ilya pictures and interact. Thursday.
• (./) Contact Carol about GSoC who may be trying to be part time.

Vincent

Alexander

Ilya

Vadim

• Find out status of new user website

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• CVPR tutorial (May 15th Early registration deadline
• Travel
• Website
  • Contribution
• GSoC: May 21 official start, can start earlier.
• New stuff
  • Inverted index
    • Need a template for this
• OpenCV world
• Box2D demo with OpenCV?
• Transparency
  • RGB and depth images of bottles
• Vincent is at ICRA, Victor out with stomach bug. Last week had 2 days of hollidays … so, minimal week

Minutes

• CVPR Tutorial
  • Android part is being helped by NVidia team
  • Vadim looking at iOS — porting Android demos to iOS.
    • iOS install instructions are here
• New proposed website site: http://opencv.top7y.ru/
  • Provide feedback on it
  • Want something like opencvmeetingnotes
  • Make youtube videos of functionality in new releases. For example, the creation of 2.4 is here
    • Expand opencv projects channel
• Need to fill a GSoC vacancy
• Travel
  • Ilya had Visa. Might come for June
  • Maria will get a visa in July
• OpenCV world
  • There are already 20 modules an growing
  • Nice modularlity but can make linking a pain
  • World is a module that includes all others like Frodo’s one ring Makes linking easier
    • Can be customized to contain only the modules that user wants to make it smaller
    • Will split some modules into even smaller modules to make them even smaller
      • Needs to be documented
      • Installation guide. Important especially for Windows OS
• Transparancey
  • Optimizing … sped up 3x so ~5s/frame
  • Works in clutter
  • RSS presentation (Ilya is going but paying for 50%)
• Intel likely to start getting involved
  • Application engineers from NN center

Vadim

Only 3 working days (Thursday, Friday and Saturday) this week due to Russian hollidays.
Still, here is the progress:
Bug fixing [Vadim et al]

2 bug in OpenCV 2.4 have been fixed: #1918 and #1931.

Other flavors of cv::split and cv::merge have been added (to use InputArray & OutputArray) to support vector<Mat >, as requested by Vincent.
Also 5+ bugs have been fixed by other teams at Itseez that work on OpenCV.
opencv-world [Vadim and Andrey]

The new super-module modestly called opencv_world has been introduced. It combines all the other modules selected by user and thus it’s much more convenient for redistribution, comparing to ~20 shared libraries libopencv_core, libopencv_imgproc etc. The module takes ~10-15Mb and it is linked from the same object files (their union) as the other modules, so the overall OpenCV compile time increased by just a few seconds. Big thanks to Andrey Kamaev from tegra team who came with the implementation idea and then significantly improved and fixed the initial version of the cmake script.
iOS port [Vadim]

I started playing with iOS port of OpenCV; With very minor modifications OpenCV 2.4.0 was built for iOS using the instructions from

http://docs.opencv.org/doc/tutorials/introduction/ios_install/ios_install.html

Next steps would be to port some of Android samples to iOS.
OpenCV site [Alexander + our contracted web masters]

The final draft of opencv.org has been created: http://opencv.top7y.ru/

You are welcome to check it out and provide feedback.
Welcome Evgeny!

Welcome the new member of our team, Evgeny Talanin, who joined OpenCV team last week. He will replace Alexander Shishkov. However, there is a good chance that Alexander will stay with us till the end of the month.

Meanwhile, Alexander Reshetnikov is on vacation till the beginning of June.

Anatoly

Accomplishments & statuses:
1)      OpenCV GPU
·         Fixed build under MacOS (disabled video decoding, it’s not supported). Closed tickets #1754, #1846.
·         Prepared draft of work plan for next 2 months.
Issues:

·         GPU team still doesn’t have any Kepler to test it with GPU module.
·         Some compilation errors under Mac (for static libs), to fix this need to wait until Mac Pro from our build farm will be free.
·         We only check compilation for MacOS, but never executed tests there, since we don’t have Mac with CUDA capable GPU. Regarding PCL, we even never compiled it on Mac.
2)      Background substation
·         Investigated current OpenCV CPU implementation, learned paper. (We going to port both current functionality and new VIBE).
Plans:

·         Continue work on background substation, implement first GPU version. Test if merging (using Python script) all cu-files to one reduces GPU module binary size.
·         Migration to Git. Prepare mini-training for Itseez teams.

Ilya

Transparent objects [Ilya]:

Optimized the pose estimation algorithm. A lot of optimizations were implemented e.g. caching results of computations and sharing them in pose estimations for different initial poses, computing a jacobian for inliers only, scaling an image to get an approximate final pose and then refining it on the original image, using downsampled edgels in geometric hashing.

Results (the success rate is the proportion of cases when the translation error is less than 2cm):

So the optimized algorithm works with approximately the same quality but it is 2.8x faster.

RGBD odometry [Maria]:
Experimented with mesh_object. Thanks to Vincent for resolving the problem with it.
Refactored the merged method rgbd + icp. Evaluated it on the TUM dataset again (see the graphics of translation errors in the attachment).
Results (the description of the errors can be found here: http://cvpr.in.tum.de/data/datasets/rgbd-dataset/online_evaluation):

RGBD ICP RGBD + ICP
Absolute trajectory error (median), m 0.387107 0.443915 0.066765
Relative transnational error (median), m 0.012085 0.013843 0.015936

RGBD has the smallest relative median error (frame-to-frame). But it also has one bad pose breaking the camera trajectory, so it has the high absolute trajectory error. RGBD + ICP has slightly worse relative median error, but its absolute trajectory error is much less, it does not break the trajectory. So on the TUM dataset merged approach is more stable than pure RGBD. But the dataset has motion blur and it’s the most likely reason of the one bad pose. ICP has several poses breaking the camera trajectory.

Other:
Maria has applied for the USA visa. The interview will be on July 3 (it is the nearest free time).

Action Items

Gary

• Ask Adrian ab Eduard or KuangQi
• P&G transparent items to Ilya

Vincent

Alexander

Ilya

Vadim

• Mail Eduard if he can do a project
• Ask if Mooly Eden is involved in NN contacts

Victor

Kurt

Ethan

From Last Time

Gary

• (./) Call Silicon Valley bank
• Send Ilya transparent items photos

Vincent

• Contact Bianca about fieldhouse and producing runs of schwag
• Push for travel access

Alexander

Ilya

Vadim

Victor

• Get Vincent’s comments in to website designer

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Opencv.org
• Workshop
• GSoC
• Future plans

Minutes

• GSoC
  • CODING BEGINS!
  • Eduard
  • Repo for code .. Alexander says we’ll make one in a couple of days using Git
• Tutorial
  • https://sites.google.com/a/itseez.com/mobile-vision-with-opencv/
  • Get this linked to CVPR site
  • Vadim has Apple developer account, working on iOS
    • Port samples to iOS
• OpenCV.org
  • Prototype website:
    • http://opencv.alekcac.webfactional.com/
    • Needs answers and wiki
  • donations etc for the site
  • Will use Citi
  • Get money in for email address
  • Documentation
  • Google is $50/year per account
    • Itseez expense for now?
    • Go with Google or create our own server. Google easier with groups
      • Transition to paid account later
    • Notifications can be done with google groups
    • need account to log in
• RGB+D odometry
  • Maria Kramer’s method for dense images.
  • Frame to frame odometry that is in Kinfu.
  • Playing with how to weight these
  • libmv? How much overlap. They are sparse. GSoC students on.
  • Go with Maria’s stuff and when it breaks go to sparse
  • SFM sparse package
  • Extend the stuff to more views, more camera types. No noise and noise. Missing entries.

Vadim

Bug fixing [Vadim, Alexander Mordvintsev]

closed 6 tickets. made FaceRecognizer classes derivatives of Algorithm, so that different properties can be set using the unified interface.
GSoC

The students have just started the work; there is no meaningful reports yet; the work on iOS samples started; Alexander Mordvintsev ran existing Python samples and fixed a few problems in the samples, in the wrappers and in OpenCV itself to make sure the samples run.
OpenCV site [Alexander Shishkov + our contracted web masters]

Another final draft :) of  http://opencv.alekcac.webfactional.com/. It addressed some of the concerns and requests, but not all. Still, the internal representation is pretty simple (wordpress-based) and can be easily edited by us (which needs to be done anyway). Today we had meeting at Itseez regarding the further steps.
Other than that, wiki has been partially converted to redmine format: meeting notes (https://github.com/opencv/opencv/wiki/Meeting_notes), Changelog  (https://github.com/opencv/opencv/wiki/ChangeLogHistory)

Anatoly

Accomplishments & statuses:
1)      OpenCV GPU
·         Tested OpenCV GPU 2.4 compilation for MacOS manually (since buildbot works with trunk), fixed probable compilation errors (which one user reported, but we can’t reproduce), closed ticket #1901.
·         Performance tests framework.
o   Yes, we can use them to measure CPU-GPU speed-ups. To do this, we’ve implemented 90% of GPU test cases using CPU functions (in order to compare them using the python scripts). Added tests cases for 24 uncovered functions.
o   Going to finish remaining cases, prepare complete log and send to NVIDIA tomorrow. There are some surprising results for us.
3)      Background substation
·         Implemented first GPU version of FGD Foreground detection (~ 5-6x speedup).That is old OpenCV algorithm, not VIBE.
4)      Other
·         Hiring. Interviewed one candidate in ‘PCL for Tegra’ project. He is good, probably will hire him.
·         Spend some time for learning GTC videos, posters, etc.
·         Prepared materials for Git mini-training, plan to conduct it when Git hosting is ready.
Plans:

·         OpenCV performance testing, investigation, optimization.
·         Continue with FGD Foreground detection on GPU (test implementation on different cards and different OS, try to optimize implementation).
·         Recording BodyBarts video for NVIDIA (after fixing acquisition problems). Continue work on Pseudo Convex Hull.

Ilya

Transparent objects [Ilya]:

Finished the first draft of the camera ready paper for RSS 2012. Created a private git repository for it so we can easily collaborate on the paper.

Searched and studied new papers related to transparent objects. There are 2 interesting papers:

1. Zhong Lei, K. Ohno, M. Tsubota, E. Takeuchi, and S. Tadokoro. Transparent object detection using color image and laser reflectance image for mobile manipulator. IEEE International Conference on Robotics and Biomimetics (ROBIO), December 2011. Light detection and ranging (LIDAR) data are used to segment unknown transparent objects: highlight spots from a RGB image are used to find candidate areas of transparent objects and then the GrabCut segmentation algorithm is applied to a depth image and a laser reflectance intensity image to compute the final segmentation. The experimental setup consist of a 2D LIDAR device, a registered RGB camera and a pan-tilt unit so it is more complex than Kinect which we use.

2. Meriaudeau, R. Rantoson, D. Fofi, and C. Stolz. Review and comparison of non-conventional imaging systems for three-dimensional digitization of transparent objects. Journal of Electronic Imaging, May 2012. This paper covers different approaches to scanning transparent objects which were published after the excellent previous review on this topic: I. Ihrke, K.N. Kutulakos, H.P.A. Lensch, M. Magnor, and W. Heidrich. State of the art in transparent and specular object reconstruction. In EUROGRAPHICS 2008. Results are still the same: there is no robust and reliable technique to scan transparent objects in general case.
Other:

Bought flight tickets to/from the USA. I still have my passport in the UK visa center but they officially guarantee that I have at least 90% chance to get it the next week and I have 98% by statistics in March.
RGBD odometry [Maria]:

Installed more dependencies of the ecto capture pipeline. So now all the capture scripts work for me. Captured a couple of objects using an orb template to practice with the pipeline, got their meshes in the browser.

Replaced the existing pose estimation in the pipeline by rgbd odometry. Saw the first results and it seems I applied transformations in the wrong order somewhere, investigating this now. A warped image was stable in that experiment and it means rgbd odometry gave accurate poses.
Other:

The interview for the visa has been rescheduled for June 22.

Action Items

Gary

• Check with Carol if Eduard is in
• Go to guy who manages tutorials and get the link on the CVPR page above
• Try registration on our site

Vincent

• Send email summarizing libmv work

Ilya

• .

Vadim

• Look into email accounts: Google or some other? Commercial or some other?

Victor

• Make sure answers and wiki are connected to the new opencv website

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Workshop
• New website
• RGBD demo

Minutes

• CVPR Vision on mobile devices workshop
  • Note the prerequisite section
• Alexander will come to CVPR
• Intro: how to build ,step by step introduction. Android is there, making iOS tutorials too (inpainting, face detection)
• What should the mobile app be?
  • Face detection
    • might not be real time on mobile. Instead 2 threads: Detect and track
    • For iOS, we can use standard face detection code because it supports pThreads
    • Either that or do ORB + some homography
• We’ll order schwag site
• Schwag store=
• OpenCV website, launch is ~ Early June
• Model capture
  • Getting to be high priority
  • Get good odometry

Vadim

[Bug fixing; OpenCV team et al]

18 tickets have been closed; 7 of them – by OpenCV team.

Most of them are related to documentation and various build/install issues.
[New functionality: Vadim]

Now all the OpenCV data structures can be stored to text strings and restored back

//==== storing data ====
string buf;
{
FileStorage fs(".xml", FileStorage::WRITE + FileStorage::MEMORY);
fs << "date" << date_string << "mymatrix" << mymatrix;
buf = fs.release();
}
//== reading it back ==

{

FileStorage fs(buf, FileStorage::READ + FileStorage::MEMORY);

fs[“date”] >> date_string;

fs[“mymatrix”] >> mymatrix;

}

Such functionality can be useful for mobile platforms and in large-scale systems that use databases to store data instead of regular files.
[Build infrastructure: Evgeny, Alexander]

Updated our build environment to OSX 10.7, new VMWare, new Linux distros (including Ubuntu 12.04LTS), added parallel MinGW+TBB build, moved some builds to different machine, so now it’s much, much faster. The ultimate goal is to have “build-test before commit” feature to make the trunk version of OpenCV always more or less stable.
[opencv.org, code.opencv.org: Alexander]

a few tweaks to opencv.org have been made to opencv.org: http://opencv.alekcac.webfactional.com/

created SVN repository for GSoC 2012 projects at code.opencv.org. Unfortunately, git still requires more time to configure it properly.
[iOS port: Eduard, Vadim]

Eduard Feicho now officially started working on the project (thanks to Gary!), he setup all the environment and built OpenCV for iOS using the already supplied with OpenCV scripts.

I created a simple app that reads video from camera and processes it using OpenCV function (laplace); will now proceed with something more complex, e.g. face detection.

Anatoly

Accomplishments & statuses:
1)      OpenCV GPU
·         Reimplemented all GPU performance test cases with CPU functions in order to compare CPU vs. GPU. Complete log was sent earlier.
o   Minor improvements in OpenCV performance testing framework for functions than run fast.
o   Spend some time for investigation the results, can present outcome in separate email or during Skype meeting.
o   Marked items in the log where NPP is called.
·         Tested GPU module with CUDA 5.0, fixed some problems (LUT), closed issue #1982 (CUDA5.0 compilation errors). We are ready to release 2.4.1, waiting when Tegra team will be ready.
2)      Background substation
·         Fixed build problems for old architectures.
Plans:

·         Continue with FGD Foreground detection on GPU (optimization).
·         Recording BodyBarts video for NVIDIA.
·         Continue work on Pseudo Convex Hull.

Ilya

RGBD odometry [Maria]

Found the bug mentioned at the last time in my initial integration of the RGBD to the ecto

capture pipeline. Made experiments to scan some models using the RGBD, see an example in the

attachment. Tried two ways of scanning: 1) static scene + moving Kinect and 2) rotated table with

an object + static Kinect. In reality it is hard to provide an ideal static scene for the whole frame

(e.g. due to other people in the room). So the RGBD was not stable in both cases and its other

results were not so good as the attached example. To satisfy this condition of the RGBD we may

run it on pixels of a table with an object only and use the second way of scanning.

Tried to get a mask of a table with an object (to pass it to the RGBD) using PlaneFitter for each

frame. But PlaneFitter returns unstable axes (rotation) if there is even a relatively small object on

the table (it is unexpected behavior of the fitter). Moreover, it slightly slows down the capturing.

So to get the stable mask the ORB-based poser is run first. The RGBD processes pixels of this

mask only which provides a stable part of a scene.

Experimented with the RGBD working on the stable part of a scene. It still does not give accurate

models because it accumulates frame-to-frame errors of a camera pose. Investigating the reasons

of the errors now.
Other:

Found and fixed a bug in ecto_opencv. This bug produced the incorrect ORB-based poser after

updating the ecto_opencv.
Transparent objects [Ilya]

Revamped the draft of the camera-ready paper for RSS.

Prepared a poster about our work with transparent objects.

Adapted a paper about our previous approach and submitted it to a conference.

Started to refactor the transparent_objects module.
Other:

Ilya will go to the USA this weekend. He will leave the office for ~1.5 month:

June 02 – June 15   Preparing the demo at Willow Garage, Menlo Park, USA

June 16 – June 21   Attending CVPR as a visitor, Providence, USA

June 25 – June 29   Participating in the BMVA/EPSRC Computer Vision Summer School (site), Manchester, UK

July 2 – July 6      Participating in the Microsoft Research PhD Summer School (site), Cambridge, UK

July 9 – July 13    Presenting our paper at 2012 Robotics: Science and Systems Conference (site), Sydney, Australia

July 14 – July 17    Vacation, Sydney, Australia

Action Items

Gary

• Call Secretary of state in California to see if we can expedite a previously filed corporate filing.
• Call Cat to give here a “heads up”
• Schwag sites

Vincent

• Check out purchase of usb keys with data
• Try to get cash from Willow

Ilya

• .

Vadim

• .

Victor

• Check out purchase of usb keys with data
• Find if groups are possible with a free account
  • else have to pay for a couple of accounts.

Ethan

• .

From Last Time

Gary

• Check with Carol if Eduard is in
• Go to guy who manages tutorials and get the link on the CVPR page above
• Try registration on our site

Vincent

• Send email summarizing libmv work

Ilya

• .

Vadim

• Look into email accounts: Google or some other? Commercial or some other?

Victor

• Make sure answers and wiki are connected to the new opencv website

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Minor release
• Website
• Workshop
• Udacity
• RGBD
• Transparent progress

Minutes

• Minor release started. 2.4.1 is out, 2.4.2 ~July, 2.4.3 in August, 2.4.4 ~Nov
  • Pretty much automated procedure, tegra team helps so we can get out more continuous releases
    • packages for linux, windows, mac and android
• Website
  • Should be out this weekend in Beta
  • Fixed some security permissions
  • Try to get non-profit corporate filings done [Gary]
• Workshop
  • There will be foils for Android part
  • Vadim on foils for iOS part
  • Preparing T-Shirt and USB key. Probably 100 copies
  • Victor is out. Vadim June 14. 15th on site. Vincent and Ilya Sat. Morn.
• Started to talk with Udacity for possible OpenCV courseware.
  • UCSD reading group.
  • Vision-class.org
    • Also Malik’s https://www.coursera.org/course/vision
• OpenCV.org Foundation could sponsor improvement of vision pages on wikipedia
  • See the ideas google docs for more
• RGBD module [Vincent and Ilya]
  • fast normals
  • fast plane finding
  • ICP
  • Code to clean up NANDS “patch nands”
• Object capture is a priority going forward
• Transparency
  • Reproduced results from the Summer onto the robot
    • About 5 secs/frame
• GSoC Reports
  • 2 students on iOS
    • 4 projects using camera etc Want to improve before CVPR
  • Alexander working on Python samples
    • Documentation: Every example should have a help that tells what it does and how to call it
    • Deva Ramanan: Parts based code Hilton port to objdetect module

Vadim

[OpenCV 2.4.1: Alexander Shishkov and Andrey Kamaev, OpenCV team]

The first stabilization release in the 2.4.x series is out. Here is the complete ChangeLog:

The GPU module now supports CUDA 4.1 and CUDA 4.2 and can be compiled with CUDA 5.0 preview.

Added API for storing OpenCV data structures to text string and reading them back:

1 //== storing data ==

2 FileStorage fs(“.xml”, FileStorage::WRITE + FileStorage::MEMORY);

3 fs << “date” << date_string << “mymatrix” << mymatrix;

4 string buf = fs.releaseAndGetString();

5

6 //== reading it back ==

7 FileStorage fs(buf, FileStorage::READ + FileStorage::MEMORY);

8 fs[“date”] >> date_string;

9 fs[“mymatrix”] >> mymatrix;

cv::calcOpticalFlowPyrLK now supports precomputed pyramids as input.

Function signatures in documentation are made consistent with source code.

Restored python wrappers for SURF and MSER.

45 more bugs in our bug tracker have been fixed
[Buildbot: Evgeny Talanin]
created pre-commit script that runs three selected builds and only then allows to check the changes into the OpenCV repository.
[OpenCV sites: Alexander Shiskov]
1. some minor fixes to the prototype of opencv.org were made. The site will likely be revealed on this weekend.
2. installed and tested the new version of Askbot. It seems that all the known problems with it have been solved. It means that answers.opencv.org should be ready at the same time as opencv.org.
[GSoC 2012]

1. Eduard Feicho created 3 sample projects for iOS that will further be turned into templates for OpenCV-based applications for Apple mobile devices:

https://github.com/Duffycola/gsoc2012
- IntroCamera that shows how to use the camera/photo library (UIIImagePicker) and the video camera with autofocus (using the AVFoundation framework).

- IntroOpenGL (and LaserTowers), which should be an example OpenGL ES project.

- IntroOpenCV, which was supposed to be a hello world example for opencv+iPhone project.
the projects are now in very early alpha stage and are to be improved and cleaned up.
2. Alexander Mordvintsev created digits.py sample that demonstrates the use of OpenCV neural nets for hand-written digits recognition.

Anatoly

Accomplishments & statuses:
1)      OpenCV GPU
·         Fixed some bugs in GPU module under Windows (convertTo alignment error, compare with scalars error).
·         Fixed issue #1991 (GPU interpolation in remap, resize, warp…).
3)      Background substation
·         Tried implementation with fixed point arithmetic’s, it’s a bit faster, requires less memory, but produce less accurate results.
Plans:

·         Continue with FGD Foreground detection on GPU (optimization).
·         Recording BodyBarts video for NVIDIA.
·         Continue work on Pseudo Convex Hull.
Other:

·         We have Russian holidays next Monday and Tuesday, and working day  at Saturday instead.
·         Marina has joined us. Alexey moved to Tegra team.

Ilya

RGBD odometry [Maria]

Investigated the accuracy of the RGBD again because:

1) As it was mentioned in the last report I did not get sufficiently accurate models using the initial RGBD integration to the ecto capture pipeline;

2) The comparison with the TUM paper in February showed that our RGBD implementation gave the same errors as in the TUM paper only if I synchronized rgb and depth with a smaller time interval (0.01s) than the interval used in the TUM benchmark by default (0.02s).

So it seemed that there are unimplemented ways to improve our RGBD quality. But there was a bug in my conversion of computed camera pose to the format used in the TUM benchmark, i.e. the problem was out of the our RGBD. Recomputed the results of my previos experiments:
paper’s results	our results
RGBD	 GICP	 RGBD	 KinFu ICP (without volume, c++ version)	 RGBD + ICP

freiburg1/desk (573 frames)

translational_error.median (m)	      0.0053	 0.0103	 0.0058	 0.0066	 0.0054

rotational_error.median (deg)	              0.0065	 0.0154	 0.0059	 0.0111	 0.0055

absolute_translational_error.median (m)	       -	 -	 0.1371	 0.4439	 0.0661

freiburg2/desk (2893 frames)

translational_error.median (m)	       0.0015	 0.0062	 0.0022	 0.0077	 0.0021

rotational_error.median (deg)	               0.0027	 0.006	 0.0033	 0.0058	 0.0032

absolute_translational_error.median (m)	       -	 -	 0.2432	 2.3209	 0.2176
Сonclusions from the updated results:

1) our RGBD has accuracy close to the paper’s accuracy even we use the default parameters of both the benchmark and the RGBD (from contrib module);

2) RGBD + ICP has the smaller errors (frame-to-frame and absolute) than the our pure RGBD and pure ICP.
With more confidence in our RGBD quality I am choosing the method for the global refining the camera poses and captured model, reading the papers about this now.
Other:

Maria is on vacation for a week from June 4.
Transparent objects [Ilya]

Prepared the final draft of the paper for RSS.

Improved the poster by taking into account comments from colleagues (thanks a lot to all who provided their comments).

Continued to refactor transparent_objects module.

Helped Vincent with his implementation of the fast computation of normals.

Set up the working environment on my Willow machine and reproduced pose estimation of transparent objects on a robot with the old algorithm (without grasping yet). Thanks to Vincent for help.

Action Items

Gary

• Try to get corporate filing done

Vincent

• Get intern to add “patch nands”

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

From Last Time

Gary

• Call Secretary of state in California to see if we can expedite a previously filed corporate filing.
• Call Cat to give here a “heads up”
• Schwag sites

Vincent

• Check out purchase of usb keys with data
• Try to get cash from Willow

Ilya

• .

Vadim

• .

Victor

• Check out purchase of usb keys with data
• Find if groups are possible with a free account
  • else have to pay for a couple of accounts.

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

Minutes

• Vincent: schwag sent yesterday
• Ilya: integrated algorithm for transparent clutter. Works on the robot
• website: out this week. Everything seems ready.
• Vadim: get packages for Android and iOS. We’ll put them on the keys. 0 install: just copy. Put links to develop for both platforms and how to install device
• GSOC: iOS and Python: good progress, bugs fixed in Python wrapper. Non-local means implementation
• logging in OpenCV: nothing, look into glog.

Vadim

[iOS port; tutorial preparations: Vadim, Eduard and Charu]
last week most of the time was spent in improving OpenCV support on iOS and preparing some demos.

First of all, a script that creates opencv2.framework, an iOS framework, has been created.
It turned out that iOS simulator does not simulate ARM instructions, it’s i386 “virtual machine” with special versions

of Cocoa libraries. In other words, when you build an app for simulator, it’s i386 CPU (aka 32-bit Intel arch); when

you build the same app for real device, it’s ARM. So, we needed 2 different sets of OpenCV libraries to make an

OpenCV-based app to compile and run on both simulator and device. If we take into account that there are

already ~20 modules in OpenCV, setting all the include and library paths in an application may be really boring

and not forward-compatible thing. But now using the script we can create a “universal framework” called

opencv2.framework. It combines both versions of the library, together with the headers. It’s super-easy to add

this framework to application, just like one adds system frameworks. Xcode automatically links application with the

correct variant of the library, depending on the target.
Meanwhile, GSoC students, Eduard and Charu, did excellent job on creating new iOS samples. Right now we have

more or less good working face detection demos. One works with still images from the photo library, another one

works with live video from camera. The screenshot is attached.
[GSoC Python interface improvements; Alexander]
Alexander Mordvintsev continues work on python wrappers and samples; he added description to digits.py,

added new sample on laplacian pyramid; fixed 2 bugs in the wrappers and other part of OpenCV causing

bugs in python samples.

Anatoly

Accomplishments & statuses:
1) OpenCV GPU

Found reasons and fixed thousands of warnings about “undefined anonymous” in device code under Linux.

(need to implement copy-constructor for device functors themselves).

Resize with INTERP_ARA flag (usually used for big scale coeff, where bilinear or others behave very bad.

Algorithm example: SURF)

Added simple implementation of cv::gpu::resize with INTERP_AREA interpolation type. Average speed-up

25-65x for scale coeff = 3..5. (GTX470 vs Core i5).

Fixed bugs in CPU version of resize function related area interpolation path. Changed behavior of CPU

resize implementation according right formula for tile with not full area.

For  large scale coeffs (>10x) speed-up is very little. Working on smarter version for this case.

Spend time for dense optical flow (PyrLK) optimization ( we focues on sparse before, dense method

was a bonus)

Using shared memory for caching first image – it gives maximum 1.2x speedup in some cases.

Using texture memory for second image – it doesn’t gave any speedup.

Using float source + texture built-in linear interpolation – it gives ~2x speedup.

Video stitching application. Updated to support interface changes in opencv_stitching module. Tested

the app and sent to NVIDIA.
2) Background substation

Integrated FGD foreground/background subtraction to OpenCV (total 6x speedup).

Plans to work on VIBE
Plans:

Test new dense optical flow implementation, try some other ideas. Think how to make sparse optical

flow faster.

Implement faster resize with INTEP_AREA flag for large scale coeffs.

Ilya

Transparent objects [Ilya]:

Merged a branch with the algorithm of pose estimation in transparent clutter into the master branch.

Experimented with it on the robot and made sure it works correctly.

Documented public API of classes for recognition and pose estimation. Implemented a simple sample

how to use our algorithm and described it in the Ecto Recognition Kitchen documentation. Cleaned the

code of our algorithm, submitted the camera-ready paper to RSS 2012.

Investigated and solved the issue with too large memory requirements of our algorithm. The reason

was STL doesn’t release memory of deleted unordered_multimaps but holds it for future objects. We had

several temporary but huge hash tables in our algorithm and so memory was not released after their deletion.

Switched to using cv::Ptr to create hash tables only once and it solved the problem.

Vincent fixed integration of Ecto with the grasping pipeline after changes in Fuerte and the robot was

able to grasp a transparent object after these fixes.
Other:

Maria is on vacation.

Action Items

Gary

• Try to get corporate filing done
• Work on CVPR foils

Vincent

• Get intern to add “patch nands”
• Help with cvpr workshops

Ilya

• .

Vadim

• Work on CVPR workshop foils

Victor

• Work on CVPR workshop foils

Ethan

• .

From Last Time

Gary

• Call Secretary of state in California to see if we can expedite a previously filed corporate filing.
• Call Cat to give here a “heads up”
• Schwag sites

Vincent

• Check out purchase of usb keys with data
• Try to get cash from Willow

Ilya

• .

Vadim

• .

Victor

• Check out purchase of usb keys with data
• Find if groups are possible with a free account
  • else have to pay for a couple of accounts.

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

General Meeting

• Modular development

Minutes

• Should modules be more distributed?
  • Like linux packages
  • Keep main part small and simple as possible.
  • Download opencv core, download package. Click on cmake box and try to build. We wouldn’t stand behind ex-modules
  • Want OpenCV “brand” to mean, the code has been curated, it’s something that works. Reduce the size.
  • If OpenCV means an entity/community. If code is written for a paper, we can host if you use the library
    • Default feature detectors
• More acknowledgement for contributions.
  • Badges for contributors.
    • On discussions forums
  • Ack in doc? Like cite for paper for contributors too.
  • Physically thank top contributors
• Needs for foundation:
  • Web programmer. Web crashes, ask bot
  • Someone who takes care of community. Such as thanking contributors. Encouraging bug fixes WITH badges
• Points from
• Money
  • Rotating bill weighted by %
  • Bug fix priorities
    • levels of support
      • Phone support(?)
    • board position
    • developer control
• Infrastructure
  • Hire another engineer to fix bugs
  • Buildbot is taking full time right now.
  • 3 software engineers
• Priorities
  • SfM, Bundle Adj., RGBD, VO, Computational photography, Segmentation
  • So far, OpenCV includes classic algorithms. Policy on state of the art?

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• GSoC Status
• Website
• Foundation discussion and mission
• CVPR harvest
• FREAK
• Khoronos — Victor is now Chair! of the new Vision API group.

Minutes

• Getting OpenCV.org filed:
  • There are several services listed:
    • [email protected] ,
    • www.sierraexpress.net/sacramento-sos-expedited-filings.aspx , as well as
    • www.sundocumentfilings.com.
  • the SOS offers expedited service on their website as well but I could not follow how one was to use it.
  • The SOS website is: www.sos.ca.gov
• Mobile workshop at CVPR was standing room only. Here’s the iOS slides
• FREAK is in. Descriptor extractor but using SURF for detector
• AGAST is at Code
• GSoC
  • iOS port Eduard had an exam, but did a lot before CVPR which helped a lot for the workshop
  • Camera is working for iOS, needs to go into HighGUI
  • Simpleflow coming along
    • Single scale example with algorithmic tests is done
    • Going to multiple scale
    • Might also get a side benefit of a faster bilateral filter
• Website is good to go. Deploy any time.
  • Do a once over
• Khronos committee is now fully formed and working on a vision standard. Victor is the chair.
• Will work with Stefan Hinterstoisser for final version of LINEMOD by September
• Priorities ’12
  • iOS
  • libmv
  • More calibration improvements 2D, 3D, mix
  • Model capture
  • VO
  • optimization —
    • multi-threading on Windows and Mac using their native tools
    • Allow swapping out TBB for OpenMP for those that don’t have
    • OpenCL use wherever possible
    • Many more SSE intrinsics
    • In general, we’re moving this into single optimization class so that all code becomes the same but can specialize to architecture
      • We will then immediately comply with Khronos when that standard is finished.
  • Mobile
    • iOS (coming along nicely)
    • Computational photography

Vadim

[Tutorial about iOS+OpenCV programming: Vadim, Eduard and Charu]
Tutorial on using OpenCV on iOS has been prepared and given at CVPR’2012 in

Providence, RI. These are the slides:

http://code.opencv.org/svn/gsoc2012/ios/trunk/doc/CVPR2012_OpenCV4IOS_Tutorial.pdf.
The tutorial went well (though, because no attendees had any experience with iOS programming

and, therefore, no iOS developer account, they only experiment with iOS simulator, so not all the

samples could be run). Before ECCV’12 we should do something about it.
Eduard and Charu continue to refine iOS port of OpenCV, support for color input from camera

has been added and few bugs have been fixed.
[FREAK feature detector+descriptor]

The new feature detector, contributed by EPFL, has been integrated into OpenCV trunk, features2d module.
[Buildbot: Evgeny]

the new builders have been added: with Qt support, OpenGL, Windows with GPU;

email notifications problem resolved, now receiving information about every build

finished pre-commit script that should test working copy before committing it to trunk.

Unfortunately, since we now on SVN, the script is not very convenient to use – it’s not

quite transparent for developer. Hopefully, with transition to git it will be solved.

Ilya is traveling.

Here is update from Maria:
Cleaned up an odometry evaluation application:

Removed median filtering that was applied to image and depth. For previous RGBD

implementations this filtering improved results on TUM benchmark. But current implementation

is slightly accurate without this filtering (I suppose due to M-estimator added last week).

Found experimental uncommented code in the application. Without it RGBD has less errors on

TUM benchmark.

There are no updated RGBD results, since Maria is on sick leave since June 21th.
Also, Maria came to Moscow for interview and could not get visa :(

Marina

Accomplishments:
OpenCV, Vlad.

ViBe Background Subtraction. Fixed ViBe implementation for large resolutions.

MOG. Implemented two CUDA variants of algorithm.

Integrated all algorithms into the gpu module. Added accuracy and performance tests, documentation,

samples. The performance results are below.
OpenCV, Marina.

Read Anton’s gem about GPU Haar cascade, investigated possibilities of LBP cascade

implementation on GPU.

Added ability to read new xml format of OpenCV classifiers to GPU classifier.

Implemented draft version of LBP classifier on CUDA.

Integrated scan in device layer of gpu module.

Issues: None
Plans:

Add new demos for background subtraction to GPU demo pack.

Test and optimize LBP cascade classifier implementation.

Other:
Vlad will be on vacation starting from July 2

Background subtraction performance results for video with resolution 768×576
VIBE CUDA     MOG CPU      MOG CUDA     MOG2 CPU     MOG2 CUDA

Avg. FPS      2140         50           1200            135         1500

Ilya

Traveling

Action Items

Gary

• Check-in where we are globally in GSoC
• Get corporate non-profit filing done

Vincent

• .

Ilya

• .

Vadim

• Mail Kirell Benzi about AGAST code
• See about getting AGAST in http://sourceforge.net/projects/agastpp/

Victor

• .

Ethan

• .

From Last Time

Gary

• Try to get corporate filing done
• Work on CVPR foils

Vincent

• Get intern to add “patch nands”
• Help with cvpr workshops

Ilya

• .

Vadim

• Work on CVPR workshop foils

Victor

• Work on CVPR workshop foils

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• GSoC, Miderm evals in July 9 – 13
• OpenCV Foundation now established, search for “OPENCV” https://businessfilings.sos.ca.gov/
  • Issues
• Primesense
• Khronos
• AGAST, FREAK
• Model capture priority
• Workshops ECCV, ACCV?
• Cloud connections
• Python PIL connections
• Get Malik’s Grab Cut

Minutes

• GSoC Get mid term evals in 9th-13th
  • libmv moving along. Triangulation with multiple images, reconstruction
  • Python … amazon cloud example through python
    • Maybe connect him with the cascade project to train in cloud
  • Simple flow, single scale. Victor will sync this week.
• Foundation is established
  • Interface to http://opencv.org/
    • Alexander Shishcov
  • Paypal
  • Schwag
  • Get starter funds
• Primesense for OpenNI 2.0
  • How is OpenNI supported. We just need to link to the correct one.
• Khronos
  • Victor is chair, survived back wounds. Next meeting August face to face
• AGAST — feature part of BRISK
  • Best of ORB and BRISK paper Intern on “yet another binary discriptor” but on better distance function
    • OpenCV foundation could host a mini-conference/hackathon and come out with a “best of breed”
• Model capture
  • Ilya’s kin

Vadim

[OpenCV 2.4.2: Tegra and OpenCV teams]
The next OpenCV update is almost released. The code has been frozen and the packages are being created at the moment.

Here is the redmine page for the release:

http://code.opencv.org/projects/opencv/versions/6
in short, the new release will get about 45 bug fixes, OpenCV service for Android, OpenCV framework for iOS, comprehensive documentation and samples for the FaceRecognizer, some more GPU optimization etc.
Big thanks to Tegra team who actually did most of the work on preparing the release.
[OpenCV site: Alexander Shishkov]

Finally, http://opencv.org and http://answers.opencv.org are now running!
[Contributions]

pretty interesting background subtraction algorithm GMG:

“Visual Tracking of Human Visitors under Variable-Lighting Conditions for a Responsive Audio Art Installation”, A. Godbehere, A. Matsukawa, K. Goldberg

has been contributed by the algorithm authors, along with the sample and tests. The screenshot is attached.
[GSoC 2012]

Charu and Eduard continue to work on iOS samples.

Charu started working on camera calibration + augmented reality demo for iOS. Camera calibration part has been written and is being debugged now.

Eduard is working on improving and accelerating VideoCameraController class that captures video from camera on iOS.
Alexander Mordvintsev continues to improve digits recognition Python sample – with better feature selection and tricks from CVPR’2012 paper:

R. Arandjelovic, A. Zisserman

“Three things everyone should know to improve object retrieval”

the performance is now much better and SVM trains much faster.

Descriptions to 2 other Python samples have been added.
[Buildbot: Evgeny]

binary compatibility between 2.4.1 version and 2.4 branch is now regularly tested. Overall verdict is ‘compatible’ but still no public webpage exist

added iOS framework builder, it works and seems to be good for using in XCode projects;

configured Android builds to be run simultaneously, and Android tests to be run in parallel with any non-Android builds. Total duration of build queue on Bigmac reduced by ~1.5 hours;

modified Ubuntu x64 QT builder: installed number of video I/O libraries, upgraded Qt to 4.8.2;

minor activities: run tests via run.py, correct removing old MegaPack builds result etc.

Anatoly

Accomplishments & statuses:
1)      LBP Face detection
·         Implemented simple GPU pipeline for LBP classifier. Going to measure performance and analyze bottlenecks.
·         Fixed a few bugs in cascade loading function.
2)      OpenCV GPU
·         Improved CPU/ GPU similarity of results for nearest neighbor interpolation (resize, remap, warpAffine, etc.). Merged to 2.4 branch.
3)      Background subtraction
·         MOG implementation was reintegrated in 2.4 branch (previous merge was with mistakes)
Plans:

·         Complete GPU implementation of connected component function for classifier hypothesis filtration. Measure performance and bottlenecks. Think how to make it faster.
·         Possibly try another idea for cv::gpu::resize with INTERP_AREA flag.
·         Roman is going to learn Android/Tegra 3 this week and PCL bodyparts sources.
Other:
·         Vlad is on vacations during whole July. Anatoly came back from vacations and have not answered all emails.
·         We have meeting this Wed, don’t we?

Ilya

RGBD Odometry [Maria]
Closed TODOs with my code in ecto_opencv/rgbd:

- Refactored and converted odometries to cv::Algorithm. Now there are 3 classes for RGBD, ICP and merged approach.

- Added doxygen doc.

- Added tests for 3 odometries.

- Added sample for evaluation on TUM benchmark.
OpenCV fixing for incoming 2.4.2
Fixed #1764, #1834, #1879, #2043, #2082.

Added test for opponent brief.
Other:

Ilya attended BMVA Computer vision (site) in Manchester (UK). He moved to Cambridge (UK) to attend Microsoft Research PhD Summer School 2012.

Action Items

Gary

• Send out reminder on midterm evals for GSoC people
• Contact Thrun about .org

Vincent

• Zisserman Contacts
• Ping primesense back about using OpenNI 2.0

Ilya

• .

Vadim

• Get example of how to use cloud for training cascade for my student

Victor

*

Ethan

• .

From Last Time

Gary

• Check-in where we are globally in GSoC
• Get corporate non-profit filing done

Vincent

• .

Ilya

• .

Vadim

• Mail Kirell Benzi about AGAST code
• See about getting AGAST in http://sourceforge.net/projects/agastpp/

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• AVG
• Foundation progress
  • New website publications
• Priorities
• GSoC
  • Student evaluations need to be turned in
• MOSSE

Minutes

• Install AVG — no.
• OpenCV 2.4.2 has been released on July 4th:
  • Change log Release 2.4.2
• GSoC
  • “Nice tracking video (Youtube)”: http://www.youtube.com/watch?v=FirtmYcC0Vc
  • Student evals in
  • Vadim today or tomorrow
  • Python going pretty good
  • iOS will have an app showing many demos
• Foundation
  • Still waiting on banking keys, codes for full access. As soon as we have this, make more noise
• Priorities
  • MOSSE
  • Eulerian Mag
  • Mobile acceleration. TBB
• Taking a theoretical look at descriptors …
• Tracking
  • Look at tracking code from last year’s GSoC, some of it was good, get it in
  • Contrib or Video
  • Meanshift is in contrib
  • Meanshift alg is in video
  • VO 35% faster
• Segmentation engine
• Matlab
  • Mexopencv
  • Headers only to make it easy, Matlab to Mat
• Release
  • 2.4.2 Release is out now
  • 2.4.3 End of Summer
  • 2.5 end of year
• Hackathon developer guide.
  • Sept
• Khronos F2F in July in Texas

Vadim

[OpenCV 2.4.2: Tegra & OpenCV teams]

OpenCV 2.4.2 has been released on July 4th:

https://github.com/opencv/opencv/wiki/ChangeLog
[GSoC2012, iOS: Eduard & Charu]

Eduard and Charu continue work on the OpenCV for iOS samples.

We decided to prepare free apps by the end of the project: “OpenCV Demo” and

may be an “OpenCV Image Effects” and put them to AppStore, in order to

popularize OpenCV.
Here is screenshot of the current version of “image effects” demo.

The idea is to make “image effects” runnable on simulator, so that people can play

with iOS and OpenCV without developer account or real ios device. “OpenCV Demo”

however, will be video based demo, demonstrating face detection, elements of AR, video

effects etc.
[GSoC 2012, Python samples: Alexander]

created very nice sample for planar object tracking using ORB features:

http://www.youtube.com/watch?v=FirtmYcC0Vc

made the how-to on using PiCloud

started experiments with Kinect (using OpenNI-MS bridge, but this could possibly

be replaced with highgui at some point)

[Python bindings: Eric Christiansen and Vadim]

A part of contrib module is now wrapped in Python too; it means that chamfer matching,

variational stereo correspondence, tone mapping and, most importantly, face recognizer

are now available for Python users.
[Buildbot: Evgeny]

done first tests of the pre-commit script and received feedback from the Tegra team.

wrote proof-on-concept hook for git to run pre-commit (actually, pre-push) testing when

we move to git.

started installation of a new, much more powerful redmine server at Hetzner.de, where

we create opencv git repository. Optimistic estimate of the transition time – 1-2 weeks.

Anatoly

Accomplishments & statuses:
1)      LBP Face detection
·         Improved GPU version for LBP classifier cascade (Improved register usage in

LBP classifier, fixed compatibility with early CUDA architectures).
·         Implemented GPU version of hypothesizes grouping (Union-find connected components)

to eliminate remaining GPU/CPU data movement in cascade pipeline.
·         Current version is up to 2x faster than TBB optimized CPU version

(GTX470 vs. Intel i5-2500k 3.3Ghz quad core)
Plans:

·         Continue work on LBP optimization, write custom kernels to support for deep trees.
2)      OpenCV GPU
·       Fixed #2130, #2147, fixed codecs initialization bug.
·       Added destination matrix size checking in integral and resize functions to avoid extra

GPU memory reallocation.
4)      Background subtraction
·         No update
Other:
·         Learned DTAM paper as candidate for implementation

(http://www.youtube.com/watch?v=Df9WhgibCQA)
Plans:

·         Continue LBP cascade classifier optimization.
·         PCL4Tegra: Write a naive version of the algorithm and integrate it into the Android demo app.
·         Try another optimization idea for ‘cv::gpu::resize’ with INTERP_AREA flag for large

scales (after LBP work has been finished)

Ilya

RGBD odometry [Maria]

Made some optimizations of odometries code (Mat elements access, precomputing,

conditions order, etc). That speeded up the algorithms 25-35%.
Chose the paper

Highly Accurate 3D Surface Models by Sparse Surface Adjustment by

M. Ruhnke, R. Kümmerle, G. Grisetti and W. Burgard, ICRA 2012)

for the global refinement of camera poses and point clouds for a model capture task.

Experimented with the paper’s model of Kinect depth errors due to quantization in our odometries.

There are not results yet.
OpenCV [Maria]

Simple fix #2118.
Other:

Ilya attended Microsoft Research PhD Summer School 2012. The school was starring

Antonio Criminisi, Jamie Shotton, Tony Hoare and many others.

Ilya moved to Sydney (Australia) to participate in the RSS 2012 conference and

present our work on transparent objects.

Action Items

Gary

• Set up first board meeting
• Ping Achin
• Car detector A.T.

Vincent

• Look into forming an API for a tracking module
• See what tracking code should go in

Ilya

• Present our work on transparent objects at RSS.

Vadim

*

Victor

• .

Ethan

• .

From Last Time

Gary

• Send out reminder on midterm evals for GSoC people
• Contact Thrun about .org

Vincent

• Zisserman Contacts
• Ping primesense back about using OpenNI 2.0

Ilya

• .

Vadim

• Get example of how to use cloud for training cascade for my student

Victor

*

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Adrian sez:
  

  
  
  StereoBM and StereoSGBM classes not being derived from a common base class.
  
  I realize that both of these objects take slightly different initialization variables, but there are many
  
  better ways to handle this.  In fact, most of the rest of the library uses some kind of initialization
  
  structure which is itself derived from some common base class, such that each derived stereo
  
  algorithm can be initialized by its own custom initiailizer.
  
  

• Update on foundation.
  • Paypal up
  • Board
• Credit cars or just Paypal? … $300/year cost
• Watch lists on websites?

• ASIFT
  *

Minutes

• Stereo: refactor to inherit from base class
  • Can’t do it right now due to backward compatibility for 2.4*
  • Will do in 2.5
• ASIFT
  • Can apply to other detectors: Orb, FREAK
  • Implemented in python right now, worth moving into OpenCV
• Next 2.4 release, 2.4.3 will be in Sept.
  • 2.4 focus is
    • increasing optimization,
    • better camera support,
    • GSoC for new stuff.
    • Python much improved, especially samples (good GSoC work this year!)
    • SfM
    • Mobile is much better as well
    • Folding in last year’s GSoC tracking in. Vincent cleaning it up.
    • Photo module for computational photography
    • Better modular infrastructure
  • 2.5 might start Nov. Adrian rename conventions
• Need youtube, facebook, twitter
  • Need a person for this. Board meeting issue, leverage community
  • Adrianna has left Citi. Christine as new person.
• Itseez team alternates “bug” duty to close out old bugs and deal with new to keep this much more up to date.
• Look into FAST code 5-9 as well as what we have 9-15

Vadim

[GSoC2012, iOS: Eduard & Charu]

Both students have successfully passed mid-term evaluation.

They wrote the reports for the passed part of the projects (http://code.opencv.org/svn/gsoc2012/ios/trunk/doc/).
In particular, Eduard created a draft version of iOS OpenCV demo app that

demonstrates various video effects:

[GSoC2012, Python: Alexander Mordvintsev]
Alexander has also passed the mid-term evaluation.
These is update from him:
spotted the problem with SURF features
modified the find_obj sample:
support SIFT, SURF and ORB features
BF and FLANN matching
interactively explore the matching points
added ASIFTdemo (which in fact is also "AORB" and "ASURF")
similar to find_obj demo, but with affine sampling
works well in rather difficult scenarios

[OpenCV, features2d: Maria, Alexander Mordvintsev]

Maria and Alexander (who works on GSoC 2012 OpenCV Python samples) have found some
severe problems in the latest version of SURF, SIFT and ORB. It appears that the coordinate
system (left or right) used for KeyPoint::angle estimation is not specified, and different
detectors use different ones. The guys mostly fixed the problem, added new meaningful
(in addition to simple regression tests) tests that cover the problem.

These are details from Maria:
===
1) Specified the KeyPoint::angle (in the rst doc and in the code comments).
2) Implemented new unit tests of 5 types:
- a simple test of output keypoints from FeatureDetector (validity of pt, angle and size),
- tests on rotation invariance for FeatureDetector and DescriptorExtractor,
- [additionally] tests on scale invariance for FeatureDetector and DescriptorExtractor.
3) Synchronized Feature Detectors and Descriptor Extractors according to given
KeyPoint::angle specification. Thanks to Alexander Mordvintsev for the fix of SURF descriptor.
4) The tests found some possible problems: ORB shows low scale invariance (especially descriptor),
FREAK shows low rotation and scale invariance. Opened the issue #2176 to find the reasons of this
(a problem in the tests or in the algorithms).
===
There are still some problems with scale and rotation invariance that we will look into.

[OpenCV + OpenCL: user contribution]
Finally, ocl module that lived in a separate branch for a while, has been merged into trunk.
The module contains some simple and fairly complex algorithms (like stereo correspondence,
face detection) accelerated using OpenCL. Architecture of the module is similar to CUDA-based gnu module.
Currently, the code can be classified as "good" in Linus Torvalds terminology; in other words, it compiles :)
Unfortunately, all the tests fail on my mac, since it uses CPU as the default OpenCL device for some reason,
while the module has only been tested on a dedicated graphic cards. The problem will be investigated.

[Migration to GIT: Evgeny, Alexander Shishkov]

We successfully installed Chiliproject on our new server at hetzner.de, migrated our bug tracker, wiki there
and installed a test git repository. We plan to finish migration to the new server and git this week.

Anatoly

Accomplishments & statuses:
1)      LBP Face detection
·         We continue investigation of performance bottlenecks and brainstorming optimization ideas.
·         Implemented simultaneous processing of different scales. Current version is up to 2.8x faster (

GTX470 vs. Core i5-2500k, TBB)
·         Tried the idea with jobs list which is compacted after each stage. Speed-up is 2.4×. Main reason

is low GPU occupancy at small pyramid levels.
·         After all investigations, developed (in-mind) the method with constants grid size (its description

I sent yesterday) and working on its implementation.
Plans:
·         Try to offload the algorithm to the GPU using OpenGL ES shaders.
3)      OpenCV GPU
·         Learned ‘Non local means denoising’ paper.
·         Merged documentation and android service from trunk to 2.4.
·         Background subtraction: No update
4)      Other
·         Final preparation for OpenCV migration to Git which will be done in 1-2 days.
·         Hiring activities, most likely we have a candidate to fill the second PCL 4 Tegra position.
Plans:

·         Continue LBP cascade classifier optimization.
·         PCL4Tegra: optimization ‘bodyparts’ with GLSL.
·         Previous work review for speed limit sign detection algorithms. Implementing non-local means

denoising on GPU.
·         Try another optimization idea for ‘cv::gpu::resize’ with INTERP_AREA flag for large scales (after LBP

work has been finished)

Ilya

Transparent objects [Ilya]

Participated in the RSS 2012 conference and presented the paper “Recognition and Pose

Estimation of Rigid Transparent Objects with a Kinect Sensor” by Ilya Lysenkov, Victor Eruhimov

and Gary Bradski (5 min spotlight talk + 1.5 hour e-poster + 5 min talk at NICTA headquarters).
Other:

Our paper on pose refinement of transparent objects was accepted to the largest Russian computer

vision conference GraphiCon (http://en.wikipedia.org/wiki/GraphiCon). This paper covers the algorithm

that we developed ~1 year ago before switching to Kinect.
OpenCV [Maria]

All last week I worked on OpenCV because of detected problem with the KeyPoint::angle (#2043).

There was not a strict specification of the KeyPoint::angle before (units, a range of values and coordinate

system). Feature Detectors and Descriptor Extractors have used different own specifications of an angle

and it was a potential bug in a feature detection pipeline.

1) Specified the KeyPoint::angle (in the rst doc and the code comment).

2) Implemented new unit tests of 5 types:

- a simple test of output keypoints from FeatureDetector (validity of pt, angle and size),

- tests on rotation invariance for FeatureDetector and DescriptorExtractor,

- [additionally] tests on scale invariance for FeatureDetector and DescriptorExtractor.

3) Synchronized Feature Detectors and Descriptor Extractors according to given KeyPoint::angle specification.

Thanks to Alexander Mordvintsev for the fix of SURF descriptor.

4) The tests found some possible problems: ORB shows low scale invariance (especially descriptor), FREAK

shows low rotation and scale invariance. Opened the issue #2176 to find the reasons of this (a problem in the

tests or in the algorithms).
Last week was also my turn to be on duty of OpenCV track:

done #2165, #2152, #2155, #2156, #2159, #2160, #2167, #2043,

cancelled #2151, #2107, #2174, #2172.
Plans:

Back to model capture task.

Action Items

Gary

• Contact the book guy to deposit directly to our account
• Get wire information to Vincent
• Set up first board meeting
• Get Adrian to do a google doc on naming changes
• Confirm paypal link to bank

Vincent

• Send email to Roberta about wire transfer
• Confirm paypal link to bank

Ilya

• .

Vadim

• Mail to Eugene not to lose wiki notes
• Ask them to look into moin-moin for wikis
• See about getting ASIFT warping method into Features2D

Victor

• .

Ethan

• .

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• website outage code.opencv.org
• GSoC update
• Getting Google connected to opencv.org for payments
• Foundation core board meeting this Friday
  • Statement of information
  • Needs and funding
  • Undergraduates
  • Full time Web/PR guy
  • Fund raising
  • Other legal requirements
  • Goals
  • Other corporate support
  • Android
  • PR
• New features
• Freak sample, use
• ECCV mobile tutorial
• ACCV?
• Subpixel
• Vacations
  • Gary to 13th
  • Victor next week
  • Week after next Vadim
    • August may be subject to meeting interruptions

Minutes

• http://code.opencv.org is back up!
  • We have git, mirror on git hub etc
    • Need backup to happen properly
    • Git doesn’t need this of course — we have git hub and a mirror
• We need to make a guide for developers
  • Vincent will start a documentation wiki for this, see below
• Code to opencv
  • New versions of FAST
  • BRISK port
  • Super fast normal finding and plane finding (total for both, 5ms)
• GSoC
  • iOS going: Video samples and photo examples. Planar object tracking demo. Alexander’s example from python
  • Alexander implemented MOSSE tracker. Needs to be ported to C++
  • Simple flow (dense optical flow) See Middlebury benchmarks.
    • Single and multi-scale. VGA resolution. But it’s slow right now. He’s optimizing now
  • New clustering method being developed … research but maybe it pays off
  • Have trained cascade for forks
    • Get him to collect a database of profile and semi-profile faces
      • Performance numbers. Need script to train, test and report ROC, positive, false positives etc
      • Zisserman: Good frontal face detector, need profile
  • SSE optimization Core I-7
• OpenCV needs
  • Skeleton (guarantee that the library does not decay)
    • Vadim
    • FT for tech support of opencv resources, tools. Mid-level (SW engineer)
    • Accounting
  • Advance
    • FT web/PR/contest run/outreach person
    • 2 FT would make noticeable difference (SW engineer)
    • Limited time
      • Technical write
      • Travel support
      • Marketing support
      • Contests
      • Awards
      • Education, workshop
• What are the “gets”
  • Name/PR/Good will
  • Optimization
  • Support (at some level)
    • Sell support at lower levels
    • Architectural consideration (demo applications for new platforms)
  • Influence on direction
  • XYZ summer of code
• Features
  • Resampling
  • Multi-baseline stereo

Vadim

[GIT & New Server: Alexander, Evgeny]
We finally moved to the new server (http://code.opencv.org now points to it).

The migration took a bit longer than we expected, because of the differences between Chiliproject and

Redmine and other subtle nuances.

Also, because Redmine themes are not fully compatible with Chiliproject’s, we did some rough customization

of the interface.

We already tested various features, including the bug tracker, wiki, git and everything’s working well.
[GSoC Python: Alexander Mordvintsev]
Alexander continues work on tracking samples in Python. Here are the details from him:

MOSSE tracker http://www.cs.colostate.edu/~bolme/publications/Bolme2010Tracking.pdf

I successfully managed to implement MOSSE correlation based tracker; it was compared with some videos

http://www.youtube.com/user/bolme2008

from the paper’s author and I’ve got very similar results. It is quite robust and fast. The code is still need to be polished. S

ee issues section also.

notes on ASIFT

Not actually a progress, just a case of unexpected performance of ASIFT sample. Once I took a few images from a

small aircraft. I tried to run asift.py on downscaled versions of these (aero1.jpg and aero3.jpg). The result is match.png.

Commsnd line was

“asift.py —feature sift-flann aero1.jpg aero3.jpg”.

I didn’t expect it to find anything in the case when I myself hardly spot matches on the grayscale versions of images.

But it did found 42 homography inliers (out of 134 matches).

Issues

I found a couple of issues with DFT implementation in OpenCV.

In order to divide and take conjugates of spectrums I used DFT_COMPLEX_OUTPUT flag for cv2.dft. It seems to leave

half of a resulting array uninitialized, when the source is real:
a = np.zeros((8, 8))

A = np.zeros((8, 8, 2))

A[:] = 123

cv2.dft(a, A, flags=cv2.DFT_COMPLEX_OUTPUT)

print np.sum(A == 123) # expect everything to be zero, but some values are uninitialized
This is not a problem, when transforming back with DFT_REAL_OUTPUT flag, but gives me lots of warnings when I

manipulate spectums with numpy (overflows and invalid values). I’ll add this bug to a tracker when it becomes available.
Also I found, that in theory I could use CCS-packed spectrum for my operations, but OpenCV misses the capability of taking

conjugates and division of spectrums, represented in this way (while it still can multiply them, even with conjugates with

cv2.mulSpectrums). Should we add this?
[GSoC iOS: Eduard and Charu]
Eduard is working on the planar object detection demo for iPhone. The main problem now is very low performance.

We will have to profile the code and see if it can be brought to realtime somehow.

Charu added several new effects to the “image effects” demo. See the attachment.
[Bug fixing: Maria]
Fixed bugs:

- the bilateral filter for uniform image of  CV_32F type,

- Lsh index constructor in the Flann.
Supported users in the answers.opencv.org. It results to several improvements

	
(automatically keeping image types the same),
	
(allowing the norm to be passed in),
	
(better tests for FLANN)
	
(better doc of cascade detection)

and a bug #2183 (Add a norm type support to the FlannBasedMatche)

[User contributions]
Grigory Frolov, intern from Intel Nizhniy Novgorod, contributed a few SSE2/SSE4.2-optimized functions for OpenCV.

It has been integrated to OpenCV trunk, sorry, I meant, to the master.

Wongun Choi, PhD student working with Caroline, contributed the patch for calculating confidence measures from HOG

people detector (the patch is for GPU-optimized version). The patch is now analyzed by the GPU team.
[Papers from CVPR 2012: Vadim]

Studied a few interesting papers from CVPR’2012 that, we believe, may be worth implementing, since they all introduce

some techniques that can be useful beyonds the algorithms, where the techniques are used. The slides that I presented

last Friday have been shared separately. For the reference, the papers discussed are:
FREAK: Fast Retina Keypoint (already in OpenCV)

Real-time scene text localization and recognition.

A non-local cost aggregation method for stereo matching

Three things everyone should know to improve object retrieval

Anatoly

Accomplishments & statuses:
1)      LBP Face detection
·         Implemented the idea with fixed grid size – speed-up is the same(~3x). it seems that we are compute

bound (IPC is near 2.0, and control flow divergence ~33%).
·         Integrated the functionality in OpenCV CascadeClassifier_GPU class. Added support of async. invocations

for LBP only (It’s possible because LBP is implemented fully on GPU).
·         Investigated approach in the paper which Shalini send us. For now we don’t see how we can apply

feature-centric approach to optimize our code.
3)      OpenCV GPU
·         Background subtraction: no update
·         Non-local means denoising: no update.
Plans:
·         Try another optimization idea for ‘cv::gpu::resize’ with INTERP_AREA flag for large scales (after LBP work

has been finished). (Marina)
·         Implementing non-local means denoising on GPU (Anatoly)
·         Start working on Connected Components labeling on GPU (Marina)

Ilya

RGBD Odometry [Maria]

Studied the chosen paper (Highly Accurate 3D Surface Models by Sparse Surface Adjustment by

M. Ruhnke, R. Kümmerle, G. Grisetti and W. Burgard) for global pose and point clouds refinement,

ready to start implementation.
Played with the OpenCV bilateral filter on a depth image. It improved results of ICP, but the quality

of RGBD and RGBD+ICP remained the same. Moreover ICP gives the smallest translation error on TUM datasets

(they contain rich 3D structures), but it still has the largest rotation error. So I tried a new scheme of

RGBD and ICP merging: we run ICP at first and then RGBD but with the pure rotation model.

This merging gives the smallest translation error from ICP and small rotation error after refinement by RGBD.

The filter works ~0.28s on a VGA frame (with similar to KinFu parameters). It’s ~4-4.5 times slower than

our RGBD or ICP odometries. We can accelerate it technically by using TBB, SSE or algorithmically by

trying the quasi bilateral filter from the CVPR’12 paper that works O(1) per pixel (thanks to Vadim for

the talk about it on the Itseez seminar).
OpenCV [Maria]

Fixed bugs:

- the bilateral filter for a uniform image of CV_32F type,

- Lsh index constructor in the Flann.
Supported users in the answers.opencv.org. This resulted to several improvements

#2181, #2182, #2184, #2185 and a bug #2183.
Transparent objects [Ilya]

I’ve finally returned to Russia to my beloved transparent objects.

Prepared and submitted a camera-ready paper “Pose Refinement of Transparent Rigid Objects With a Stereo Camera”

by Ilya Lysenkov and Victor Eruhimov which was accepted to the main conference program at GraphiCon 2012.

We got good reviews with 3 overall ratings: Strong Accept, Accept and Accept.

Re-created infrastructure used at Willow to work with transparent_objects:

updated to fuerte and the latest version of ecto.

Started to experiment with Line2D on transparent objects.

Sometimes it works quite well (see attached success.png) but

sometimes it is distracted by other gradients (failure.png). Quantitative evaluation is on the way.

Action Items

Gary

• Contact the book guy to deposit directly to our account
• Check the bank account
• Contact Google about vendor
• Get a list from J-Y of what he wants

Vincent

• Ask Roberta about transfer of money if it isn’t there (Gary to check the account)
• Start a new wiki page: core developer guide

Ilya

• .

Vadim

• See about getting ASIFT warping method into Features2D

Victor

• Send to Eugene whether they want to do a GPU version of Simple Flow

Ethan

• .

From Last Time

Gary

• Contact the book guy to deposit directly to our account
• Get wire information to Vincent
• Set up first board meeting
• Get Adrian to do a google doc on naming changes
• Confirm paypal link to bank

Vincent

• Send email to Roberta about wire transfer
• Confirm paypal link to bank

Ilya

• .

Vadim

• Mail to Eugene not to lose wiki notes
• Ask them to look into moin-moin for wikis
• See about getting ASIFT warping method into Features2D

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• board meeting
• mess in samples folder
• status bilateralFilter
• GSOC
• BRISK

Minutes

• board meeting: went well
• samples: images should not be stored in the root but in a data folder
• several pull requests
• examples of previous contributions by corporations: summer (Intel: patches using SSE, NVidia: GPU optimized face detector, Face Space in contrib, Google, OpenCL Chinese Academy of Science with AMD)
• now that the different FAST are in there, we can put BRISK
• hooks for different bundle adjuster is possible
• GSOC Vadim: add functionality and camera interface/image reading/writing in highgui. Perf test on iOS
• parallelization: support for TBB, OpenMP through a generic interface
• bilateralFilter: replace NaN with anything (max ?) and not 0 should work

Vadim

[GIT, new code.opencv.org server: Evgeny]
- set up backup script for code.opencv.org data;

- configured github and local (thanks to Andrey Senin) mirrors;
github mirror appeared to be very useful feature; in a few days with already have at least 6 pull requests! That is, this technical tool really facilitates contribution process!
[GSoC OpenCV + iOS: Charu, Eduard]
Charu added 4 more effects to image effects iOS demo: cartoonish, focus, contrast/brightness adjustment, pixellation.

Eduard integrated video camera module into HighGUI.
Another project, not related to GSoC, but related to iOS: Timnit Gebru from Stanford ported some of OpenCV performance tests to iOS, so now we can profile OpenCV there. The code is at some early stage and has not be contributed yet.
[Parallelization: Ilya L, Vadim]
Non-template implementation of parallel_for primitive function has been implemented by the new intern Ilya Lavrenov. It opens very good opportunities for us – we can have different backends for the parallelization framework (e.g. GDC on Mac, OpenMP on Linux/Windows etc.) and the backends could be “hidden” in opencv_core. To test the new engine, Ilya optimized bilateralFilter (floating-point flavor of it) and got 2.7 speedup on dual-core Core i5.
[More contributions]

Grigory Frolov, summer intern from Intel, contributed some more SSE optimizations for the following OpenCV functions: invert(), boundingRect(), cvHaarDetectObjects() (i.e. the older C implementation of our cascade detector).
Vincent contributed 5/8 and 7/12 variations of FAST. Integrated via github pull request.

Ilya

RGBD odometry [Maria]

Added support of a transformation type to all odometries in ecto_opencv (rigid body motion, pure rotation, pure translation). It was in RGBDOdometry from OpenCV contrib before, now ecto_opencv also supports this.
Implemented the building of a depth pyramid with the maxDepthDifference threshold (similar to KinFu) and more appropriate processing the invalid depth values. But it gives almost the same accuracy (improvement is less 0.2% in my experiments) and it’s slower than used before cv::buildPyramid in ~10 times (because my version has additional checks and does not use SSE). So cv::buildPyramid was remained in the odometries.
Tried to compare our odometries with KinFu ICP on the TUM dataset. It would allow us to estimate what refinement of previous frame by the volume TSDF gives for ICP. Compiled PCL but KinFu falls with a segmentation fault in the same environment where it worked in the beginning of this year. Shared my computer with Anatoly to fix the problem.
Other:

Maria will be on vacation from July 31 for 1-2 weeks.
Transparent objects [Ilya]

Accomplished several experiments to evaluate accuracy of pose estimation with Line2D when working with transparent objects (thanks to Maria Dimashova and Stefano Fabri for useful information about Line2D).
1. Simulated train data vs. real train data

Generated artificial templates for Line2D and compared results when real train data are used. The same train poses for templates are used in both cases. Results (success rate is ratio of cases when the estimated pose is closer than 3cm to the ground truth pose):

-——————————————-

|   Train data  | Success rate |

|———————-|———————|

| Real          |    64.0%     |

| Simulated     |    64.9%     |

-——————————————-

Success rate is similar but simulated train data provide much more freedom in generating templates for different scales and rotations so simulated train data were used in subsequent experiments.
2. Accuracy of pose estimation with Line2D

Evaluated accuracy of pose estimation with Line2D on a dataset of 5 transparent objects. Line2D is able to locate an object in an image correctly but it is not able to distinguish between close templates robustly so accuracy of pose estimation is low. The following experiment demonstrates this issue:

42000 simulated train templates were generated for each object. Naive sampling is not uniform in SO group so the algorithm from the following paper was used: Generating Uniform Incremental Grids on SO Using the Hopf Fibration by A. Yershova, S. Jain, S. M. LaValle, and J. C. Mitchell, International Journal of Robotics Research, 2009. These generated templates were sorted according to their proximity to poses in the test dataset and they were used for Line2D training in this order. Accuracy of pose estimation with regard to number of used templates is shown in the attached line2d.png (success rate is ratio of cases when the estimated pose is closer than 3cm to the ground truth pose). One can see that if templates corresponding to test poses are used only then accuracy is high but if we start to add other templates then accuracy degrades significantly. Test poses are not known beforehand so accuracy of pose estimation with Line2D will be low in practice. For comparison our algorithm has success rate 0.98 on this dataset.

These results show that Line2D cannot be used for final pose estimation but perhaps it can be used to generate initial hypotheses of poses so additional experiments will be accomplished to check this.
Other:

Prepared and gave a talk about our work on transparent objects to Intel Summer School students.

Action Items

Vincent

• share document about the board
• file bug report
• add .rst for FAST
• bug report for latest OpenNI (cap_openni.cpp)

Vadim

• how to check inputArray is outputArray

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• tracking library: we have MIL, online Boosting how about feature based ? (KLT and so on from libmv)
• Eric wants to know more about the wrapper scripts (so that he can help with Java and maybe get the JavaCV guys on board)
• stereo code Vadim and I saw at CVPR

Minutes

• adding feature tracking from libmv
• wishlist of things we would like to get in OpenCV to attract new developers: simple bugs, simple features
• Evgeny: bugs
• Ilya: does get emails on the build farm
• BA for Maria and Vincent: OpenCV needs interactive calibration. We have great automatic tests for calibration. We can use the Matlab toolbox for that too. Sample for calibration with a function that spits back the current calibration and the error.
• Khronos: design of pipelines
• Eric and Andrey will talk

Vadim

[GIT repository: Evgeny]

- added experimental pre-push hook for OpenCV master branch. It runs several light builders with buildbot that build OpenCV and run tests on Mac, Linux and Windows. After some testing we decided to change the strategy: run the tests after pushes and merge changes from master to the to-be-created branch “stable” nightly if all the 40+ buildbot builders succeeded.

- added iOS and binary-compliance builders.
[GSoC OpenCV + iOS: Charu, Eduard]

Charu missed this week because of rellocation.

Eduard is still working on merging highgui OSX camera capture with the newly added iOS camera capture code.
[GSoC OpenCV + Python: Alexander]

Alexander created augmented reality demos; see the video: http://www.youtube.com/watch?v=pzVbhxx6aog
[Parallelization: Ilya L]

Ilya started the work on threading geometrical image transformation functions: remap, warpAffine, warpPerspective, resize, undistort. First version gives ~2.7-3.3 speedup on Core i5 machine on FullHD images.
[More contributions]

Grigory Frolov repaired SSE code in haar face detector haar.cpp and added AVX branch. The code has not been to OpenCV yet, but it gives ~30% performance boost on some configurations. Vincent contributed documentation for 5/8 and 7/12 variations of FAST.

Ilya

Transparent objects [Ilya]
Compared Line2D with our current approach to generation of initial pose hypotheses in transparent clutter. Results (success rate is ratio of cases when the estimated pose is closer than 6cm to the ground truth pose. Note that errors are big because it is just initial poses, they will be refined by subsequent steps of the algorithm):

Adapted Line2D for pose estimation of transparent objects by using parts of our algorithm (glass segmentation and non-maximum suppression of 3D poses).
Collected new datasets with transparent clutter.
Used Vincent’s RgbdPlane from ecto_opencv to find a support plane. Now transparent_objects supports 3 methods to find a table: RgbdPlane, plane segmentation from PCL and Fiducials.
Checked Vincent’s data of transparent objects from a PR2 robot for applicability of our algorithm.
Prepared and submitted the eventual RSS camera-ready paper for publication in the RSS proceedings by MIT Press.

Plans:
Evaluate accuracy of pose estimation with regard to percentage of occlusion.

Other:
Maria is on vacation.

Action Items

• Vincent: finish BRISK

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Just back from vacations, no discussions, just reports

Minutes

• .

Vadim

[GIT repository: Evgeny]

- created “stable” branch of OpenCV repository. The branch “master”, to which we all commit, is automatically getting

merged into “stable” once buildbot successfully built OpenCV in 40+ different configurations and the correctness tests

have passed successfully.

- changed the automated testing of user commits. Now the code is put to master anyway, and after that OpenCV is run

through express testing (3 builders vs 40+ in the full testing mode). In the case any of those builders fail,

user gets e-mail notification.
[GSoC OpenCV + iOS: Charu, Eduard]

- Charu added inpainting into the “OpenCV image effects” demo for iOS. From the screenshot below we can see that

without texture generation, which more advanced inpainting algorithms include, the results look a bit unnatural,

but still the demo could be fun to use.

- Eduard finished integration of video capturing code into highgui. The API is still being refined.

[GSoC OpenCV + Python: Alexander]
Alexander created Wiener deconvolution sample.

Besides, he, our contributor Philipp and Vadim fixed several bugs in Python wrappers: ##2100, 2242, 2257.

[Parallelization: Ilya Lavrenov]
Ilya added multi-threaded processing to cv::resize; on 2-core Core i5 with hyperthreading we get consistent
speedup over 2x.

[More contributions]
Grigory Frolov, intern from Intel, contributed SSE-optimized bilateral filter. The code has not been integrated yet,
so there no performance numbers yet.

[Vacations]
Evgeny will be on vacation from tomorrow till 6th of September. Maria got back from vacation.

Anatoly

WR

Ilya

Transparent objects [Ilya]

Implemented a script to collect ground-truth poses of occluding transparent objects in semi-automatic way. Labeled

our datasets using this script.

Implemented computation of occlusion percentage that is proportion of a test object silhouette which is occluded

by other transparent objects in a test image. Implemented evaluation of pose estimation accuracy with regard to this

occlusion percentage. See evaluation.png for results (success rate is ratio of cases when the estimated pose is closer

than 2cm to the ground truth pose). These results show that pose estimation is robust to occlusion < 40% but success

rate drops to zero if occlusion > 60%. We need to use orientation of edges to increase robustness of the algorithm.

Started to integrate Fast Directional Chamfer Matching into the Levenberg-Marquardt ICP framework.

Found a big area of papers on edge-based 3D tracking that are directly relevant to our algorithm (we call this

step “pose refinement” but it is essentially the same problem as in tracking). Studied classic and state-of-the-art

works. There are real-time approaches that works with surface edges (Real-time visual tracking of complex structures

by T. Drummond and R. Cipolla, PAMI 2002) and also extensions to silhouette edges (Exploiting Occluding Contours

for Real-Time 3D Tracking: A Unified Approach by Gang Li, Yanghai Tsin, and Yakup Genc, ICCV 2007). It is reasonable

to implement these papers at some point.
Plans:

Finish integration of Fast Directional Chamfer Matching into the Levenberg-Marquardt ICP framework.

Improve results of evaluation with regard to occlusion percentage.

Collect more data.
Other:

Maria was on vacation.

Action Items

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• GSoC
• Foundation
• ECCV
• Roadmap

Minutes

• Have student’s write summary
  • Need write access to opencv.org
  • Alexander Shishkov
  • We need to put up a blog about the GSoC accomplishments
• ECCV
  • Same foilware
  • Rome U for a workshop?
  • Need travel funds in the foundation ….
  • Need to talk to Cat at Google
• Will look at trained cascade classifiers
• Roadmap
  • 2.4.3 posponed to first of October (one month since nothing much happens in August)
  • 2.4.3 will include most of GSoC results (binary compatible with 2.4)
  • 2.5 scheduled next spring
    • Some compatibility may be broken, such as micro-modules
    • Kinect fusion being ported. Since this is modular, might make 2.4.3. Will be a whole new and much faster version, that will work as a 2.5 module
• Samples
  • Make an python script to self document what functions are there
  • Hackathon for putting documentation headers for script to read
• Transparent objects
  • ICRA deadline Sept 7th. Pose estimation in transparent clutter
    • Rotation of edges for better — fast directional chamfer matching … similar to MERL paper

Vadim

[Optimization: Ilya Lavrenov, Vadim, Grigoriy Frolov]

- Ilya has finished optimization of warpAffine (2x), warpPerspective (2x), remap (3.2x) and resize (up to 12.77x for 2×2

bilinear resize) functions. In parentheses these are speedup factors for fullhd images on Core i5 machine. For 2x

decimation (i.e. the output image area is 25% of the input image area) we now have a special code (C version for now) that

does decimation. With SSE2 intrinsics it should be possible to get another substantial improvement.
- Now OpenCV makes use of native parallel for implementation on Windows (via Concurrency library, a part of MS runtime)

and MacOSX/iOS (via Grand Dispatch Central) when TBB is not available. So far, the new parallel for is used only in a few

functions (image warping and bilateral filter), but we plan to convert other TBB-based parallel for’s to the new primitive. In

order to do that, we also added cross-platform synchronization primitives (cv::Mutex class that is actually a spinlock on most

platforms).
- The patch from Grigoriy Frolov, intern from Intel, for SSE3 optimized bilateral filter has been integrated to OpenCV. The

speedup is not huge, around 20%. The code can be further improved.
Meanwhile, Maria has made a small yet important fix in bilateralFilter to make it handle NaN’s values more gracefully (which is

important for RGBD odometry)
- We fixed CMake scripts for iOS to tell the compiler to build optimized code. Previously, we apparently passed “-O0” flag to

GCC, resulting in quite slow execution. Now it’s “-O3 -ffast-math”. Together with GDC-powered parallel_for, we should get

much better performance of our samples.
[GSoC: Charu, Eduard, Alexander, others]
iOS:

- We finally integrated Eduard’s classes for video capturing into highgui.

- Image filters and video filters samples are basically ready. The final week is devoted to reports.
Python:

Alexander put all the remaining code in work to our repository. Overall, it was excellent project with about a dozen of new

exciting samples.
Non-local means:

Victor Passichenko, GSoC student supervised by Victor and Leonid Beynenson from NVidia team, has submitted implementation

of non-local means image denoising algorithm. It has been already integrated to opencv.
“Simple flow” dense optical flow algorithm:

Yuri Zemlyanskiy submitted implementation of the state-of-art dense optical flow algorithm. The GSoC project was lead by

Victor.

• iOS Port coming to an OpenCV near you soon.

Anatoly

WR

Ilya

Transparent objects [Ilya]:

Integrated Fast Directional Chamfer Matching (FDCM) into the Levenberg-Marquardt Iterative Closest Point framework (LM-ICP)

by computing jacobians using the 3D distance transform (both for silhouette and surface edges). FDCM uses orientations of edges

so it allows to get better results in clutter.
Improved filtering of outliers by considering points with high weight only.
Used estimation of edges orientations from the paper “Estimation of Curvature and Tangent Direction by Median Filtered

Differencing” by J. Matas, Z. Shao, and J. Kittler. It work with similar quality as the corresponding part of the FDCM approach

but it is faster in 3.2x times.

Collected a new dataset with 4-5 transparent objects on each image and labeled it.
See the attached plot for results of the improved algorithm (success rate is ratio of cases when the estimated pose is

closer than 2cm to the ground truth pose).
Model capture [Maria]:

Studied the g2o framework (read the paper and doc, studied its examples).

Read the paper about RGB-D SLAM that uses g2o (“An Evaluation of the RGB-D SLAM System” from ICRA 2012). Their odometry

doesn’t have anything in common with RGBD odometry which we implemented. Studied their code (available as open-source)

as a more practical and complicated example of g2o usage than most of the demos from the g2o framework.
Started experiments with the graph of poses from our odometries. To get experience with g2o I built a simple graph with

edges connecting each pose with several previous poses. But this doesn’t decrease the absolute translational error on the TUM

benchmark and makes it even larger. Studying the reason of this and g2o more.
OpenCV [Maria]:

Replaced the value that was set instead of NaNs in bilateralFilter_32f by some more appropriate value to get correct results

for depth maps (thanks to Vadim for the hint). This constant will be made the optional parameter of cv::bilateralFilter when we can

break OpenCV binary compatibility in 2.5 version (reported in the issue #2269). Switched to use original cv::bilateralFilter in

ecto_opencv (it’s possible with some preprocessing).
Other:

Maria returned from vacation on August 14th.

• Pose recognition in clutter of transparent objects.

Action Items

Gary

• Talk to Cat at Google
• ticket 2.4.3 for auto doc of samples

Vincent

• .

Ilya

• .

Vadim

• Look at trained cascade classifiers to decide on what should be included

Victor

• .

Ethan

• .

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• GSoC travel
• OpenCV on mac
• ECCV
• Still have to put up Blog
• Foundation

Minutes

• GSoC travel: wait for Gary
• OpenCV on Mac: wait for Gary. Gary should write a tutorial about it as it is pretty easy. Vadim could help as he works primarily on Mac.
• ECCV: if we cannot send people, the team needs to teach people how to present. Tutorial is on September 7th.
• GSOC: everything ended well. Very complete GSOC. What did we learn ? Next year : make sure students can work full time on it (filter out students that do not have much time). We need to keep track of what to integrate.
• Still have to put up Blog: wait for Gary
• Foundation: wait for Gary

Vadim

[Optimization: Grigoriy Frolov]

- Grigoriy, intern from Intel, has finished the summer practice at Intel. He contributed 2 more patches, for the box filter and Haar face detector with SSE & AVX optimization. The patches will be integrated into OpenCV.
[GSoC: Charu, Eduard, Alexander, others]
iOS:

The final report has been sent. Charu and Eduard wrote tutorials about using OpenCV in iOS apps. Tutorials will be included into offical OpenCV build tree and will be uploaded to docs.opencv.org.
Python:

Alexander wrote the final report/blog entry.
Object Detection Cascades:

Cascades for detecting profile faces and silverware, trained by Attila Novak, who was supervised by Gary, have been integrated into the library. The detectors work quite well (although, they are trained for specific orientation of an object, i.e. profile faces should be directed to the right, silverware items should be oriented vertically).
Simple Flow:

Yuri Zemlyanskiy – GSoC student, supervised by Victor, has submitted simple flow dense optical flow algorithm, which is not a realtime, but reportedly quite accurate algorithm. First review round has been passed and the updated, more efficient implementation, has been submitted. It will be integrated to OpenCV, video module.
[Contributions]

7 patches from users have been integrated.

Anatoly

WR

Ilya

Transparent objects [Ilya]:

Used normals to compute silhouettes instead of morphology. It is faster (overall speed-up of the algorithm is 1.4x) and can be analyzed analytically.

Computed jacobian when normals are used to compute silhouettes. Checked validity of results by comparing with a numerical jacobian. See the attached plot for results of the improved algorithm.

Started to prepare a presentation for the Itseez seminar about our latest algorithm.
Model capture [Maria]:

Continued to study g2o and experiment with it on poses from our odometry implementations. Found and eliminated the reason why my simple graph of 3D poses (with edges connecting each pose with several previous poses) didn’t give improvement last week. Some results of the camera poses refinement on TUM benchmark:
frames count	translational_error.max

(m, frame-to-frame)	absolute_translational_error.rmse

(m, the whole trajectory)
before g2o	after g2o
freiburg1_desk	573	0.032586	0.10342	0.052493
freiburg2_desk	2893	0.025508	0.29847	0.107332
freiburg1_360	755	0.057243	0.280819	0.306784
This simple graph can improve accuracy of poses for the video sequences with relatively accurate motion estimations. But it’s powerless if there are large errors in odometry. Improvement of the graph structure is in progress, e.g. a loop closure by applying Dijkstra algorithm as it’s done in implementation of RGBD SLAM [http://vision.in.tum.de/_media/spezial/bib/endres12icra.pdf]. I’m going to reproduce their implementation graph construction but adapt it for our odometries (their code is available as a package for old ROS electric). This can not solve all problems due to inaccuracy of odometry but it will increase robustness of our algorithm.

Action Items

Gary

• Talk to Cat at Google
• write a Mac tutorial to set things up (maybe with Vadim)

Vincent

• ask for a travel request for ECCV for Vincent and one guy from Itseez

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

From Last Time

Gary

• Talk to Cat at Google
• ticket 2.4.3 for auto doc of samples

Vincent

• .

Ilya

• .

Vadim

• Look at trained cascade classifiers to decide on what should be included

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• website, wiki — so that users can add information. Right now, install to Mac is pretty bad. How to install with QT? What version with what? etc
• ECCV
• New code … docs?
• Transparent
• Model capture
• FABMAP
• Mac
• GSoC visits
• Vendor datatus

Minutes

• Wiki for users? Installation on Mac is missing. Installation is getting pretty standard.
  • We need a user wiki again. Evgeny do do?
  • Duplicated obsolete information
  • Developer
• ECCV
  • OpenCV soon to be a vendor — was a mail slip up.
  • I will be there
  • Alexander Shishkov, Kirill from NVidia about Android(?) Might not come.
  • Need to cost this out and see where we stand
• New code Make sure there is documentation and examples
  • FABMAP — documentation
  • Simple flow. Short description and sample code
• Model capture
  • Loop closure — extend
• Transparent
  • Take advantage of specularities
  • Use the model fitting that exists
  • RGBD + pre pose needed.

Vadim

[GSoC: Yury Zemlyanskiy]

Finally, we’ve integrated simpleflow dense optical flow algorithm into OpenCV. It needed couple of

review rounds; and the latest version looks great (if only quite a bit slower (more than order of

magnitude?), comparing to the near-realtime Farnerback’s algorithm). Still, it would be interesting

project to implement dense optical flow benchmark in opencv test system, similar to our features2d

nd stereo correspondence test frameworks.
[Code Contributions]

• OpenFABMAP (http://code.google.com/p/openfabmap/), implementation of FAB-MAP algorithm, has been
  integrated into OpenCV. The code has been submitted by the author who agreed to change the license
  to BSD. The code includes documentation and sample code.
• a small yet useful extension to PCA algorithm that computes the number of components to retain from
  the specified standard deviation has been integrated via github pull request.
• another useful contribution came from NVidia tegra team; OpenEXR source code is put into OpenCV build
  tree, wrapped into our CMake build engine, so that highgui is compiled with OpenEXR support by default
  on several platforms (Windows, MacOSX, Android).
• parallel version of the retinafilter, a part of OpenCV contrib module that was contributed and is being
  maintained by Alexandre Benoit, has been created by Alexandre.

[Testing: Ilya Lavrenov]
More strict tests of image warping functions have been created by Ilya Lavrenov. The tests for resize,
warp affine and warp perspective have been
stabilized, remap test is in progress.

Marina Kolpakova

Accomplishments & statuses:
2) Image Denoising (Anatoly)

• Fixed CPU/GPU results differences in bilateralFilter. Speedup is
  20-35x (C2080, i5-750, TBB, SSE optimized).

3) Hough Transform (Vlad)

• Implemented a GPU version of the generalized Hough Transform for
  the translation, translation + scale and translation + rotation
  cases (6-10x speedup).

4) Pedestrian Detection (Marina)

• Prepared soft cascade (same as cascade in the original paper).
• Implemented reading, writing, and XML representation for soft
  cascade.
• Implemented a detector that uses this type of cascade.

Plans:

• Investigate the OpenNI stability problem. (Roman)
• Implement a GPU version of the generalized Hough Transform for
  translation + scale + rotation case (fast version). (Vlad)
• Test detection rate; integrate ICF (Integral HOG-like channel
  features in the training application) features. (Marina)
• Get acquainted with Android architecture and the Kiwi library.
  (Andrey)

Other:

• Anatoly is on vacation until 2012-09-17.
  

Ilya

Transparent objects [Ilya]

Finished the presentation about our algorithm and gave a talk about it at the Itseez seminar.

Thanks to all for activity, questions and feedback.

It helped to understand how to present the algorithm and write a paper about it.

Started to write a paper about our latest results.
Model capture [Maria]

Added to the graph construction several steps/features from RGBD SLAM

[http://vision.in.tum.de/_media/spezial/bib/endres12icra.pdf]. Now the graph structure is the following:

- each node is a frame pose,

- 3 types of edges for each node:

1) edges connecting the pose with several previous poses,

2) edges to several keyframes (if there are close such ones)

3) edges to nodes that are found by Dijkstra algorithm (if there are close such ones; excluding edges from 1) )

But results of such more complicated graph are similar to the results of last week. The possible problem of this is

that our odometries are less stable to a large transformation between frames than the method from the paper (I’m not

sure that we have a chance for a loop closure with such sensitivity to the difference between frames). I’m investigating

this and the ways to improve the graph results (e.g. initialization of our odometries by a pose from features2d + 3d-3d).
Other:

Maria took two days off for personal reasons (Aug 30, 31).

Action Items

Gary

• Send note to Vadim about creating a user wiki (with log in and password)
• See about forwarding old site to new site
• Cost out ECCV
• Send Victor an invite letter for GSoC Mentor summit. Date + week past

Vincent

• .

Ilya

• .

Vadim

• Update tutorials
  • Make sure we have installation on mac

Victor

• Talk to NVidia team about giving tutorial
• Cost out ECCV

Ethan

• .

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Vincent: sparse matrices. Hilton told me that sparse matrices are barely supported (you cannot add/multiply them). Is there a plan to support them, maybe using sparsuite?
• Should the website still point to Yahoo groups?
• Google analytics on our sites
  • Google ads?
• OpenCV retrospective and going forward
  • Want to document what’s changed in 6 months in terms of
    • infrastructure (build system, modules, bug fixes, new websites, downloads, git …)
    • speedups
    • functionality, including new GSoC things, transparency
  • Going forward: new areas:
    • SfM, VO, VSLAM
    • Make a case to M and S
    • V needs.
    • Robot specific OR
• ECCV plans, expenses
  • Rome visit
• Adrian on devzone
• Forum … answers opencv.org
  • Forum for answers … new algorithm
  • Open a forum for undergrads
• Ilya paper for ICRA

Minutes

• Retrospective. Want to prove utility. Plane detector
  • ROS stuff, output
  • OpenCV past 6 month. Prepare talk
    • 2 researchers, Ilya and Maria.
    • Website, git, bugs, optimization, new functionality, GPU, Android, iOS
    • GSoC, Maria odometry being used
  • Good for general robot support in new application areas: research of course, health, retail, lodging
    • Slam, object recognition, arm calibration given cad model. Prepare a generic talk for this. 160K downloads/month
      • operating doc summary
      • MC&S. Project, tasks: health:
        • Suggestions grasp objects better, interact with people
      • L
      • S Navigation, obj rec. Need lighter weight
• Google analytics. Docs 1000 new users/day
• ECCV
  • Gary stay with Vincent
  • T shirts, stickers
  • Nadri also at Rome
  • Sefano Fabri
• Ticket is opened about sparse matrices

Vadim

[Bug Fixing]

12 tickets in OpenCV bug tracker have been closed. It includes crashes in the core and imgproc modules, in Python bindings, errors

in documentation and build scripts etc.
[Tests]

Ilya Lavrenov has nearly finished work on the new tests for image warping functions: remap, warpAffine, warpPerspective, resize.

The test for resize with area interpolation is still in progress.

Recently, Ilya has optimized those functions, and we plan some further optimization, since they are critical. And for that we need

more strict tests, which Ilya did.

*Roman Donchenko (GPU) *

Accomplishments & statuses:
2) Hough Transform (Vlad)

• Finished with the Generalized Hough Transform and integrated the
  code into OpenCV. There are two implementations:

o Ballard for the translation, translation + scale and translation +
rotation cases.

o Guil for translation + scale + rotation case.

• The GPU implementation is 6-15x faster than CPU with TBB (Tesla
  C2050 vs Core i5-760).

3) Pedestrian Detection (Marina)

• Added support for shrinkage to OpenCV soft cascade.

• Investigated Benenson’s demo.

• Implemented ICF (Integral channel feature) in the OpenCV traincascade app.

• Collected some pedestrians datasets for training.

• Issue: datasets needed for training cyclist and motorcyclist cascades.

Plans:

• Add depth visualization and recording to the demo app; ask libusb
  maintainers for advice on the data loss. (Roman)

• Continue learning the Java Native Interface and get acquainted with
  the Kiwi library. (Andrey)

• Try to improve performance of several primitive functions (ex.
  pyrDown, pyrUp). Investigate Super Resolution algorithms. (Vlad)

• Continue with training. (Marina)

Other:

• Anatoly is still on vacation until 2012-09-19.

Ilya

Model capture [Maria]

Read several papers about robust back-end for pose graph SLAM because I did not get improvements from graph

optimization for some TUM datasets last week ago (I suppose the reason is outlier and close to outlier edges). The

interesting paper was “Towards a Robust Back-End for Pose Graph SLAM”, ICRA 2012. Conclusion: outlier edges is

very hard problem having a particular importance for a long-time complex trajectory, so we don’t plan to solve it

for our task now. At first we will suppose that the odometry does not break a trajectory (uncritical number of outliers)

and we have special kind of trajectory (one turn of/around the table).
Implemented a masker of the train images to find the pixels of the table with the object. The reason is that our

odometries suppose stable scene, so we run them only on masked area. The masker is used Vincent’s RgbdPlane

(that copes very well with its task) and pcl convex hull and prism.
Experimented with the last weeks pose graph on some old TOD-like base (turning table with fiducials). Tried all

our odometries on it. The ICP and RgbdICP was not stable for such kind of scene (large plane of table) whereas Rgbd

works well even if there is relatively poor texture on the table (circles grid). The simplest structure of the graph

showed the more accurate results on the base (one edge to the previous frame and loop closure with the first frame).

See example of results in attachment: before and after graph optimization for the cup; only after graph results for

other objects. Notes: 1) circles grids existing in the images are not used, they are only texture; 2) strange pink points

around the table are my hands.
Transparent objects [Ilya]

Finished the first draft of our paper and sent it to Victor for a review.

Cleaned the code used in the paper experiments.

Action Items

Gary

• Make some foils and share with Victor
• Write Stefano to set dates for Rome

Vincent

• Figure out a way of getting the number of clones from github talk also to Vadim
• Set up Victor and Ilya talks
• Schwag store (password with capital words and numbers).

Ilya

• .

Vadim

• Figure out infrastructure, optimization, bug advances, build, modules, GSoC stuff over last 6-9 months
• Figure out a way of getting the number of clones from github talk also to Vincent
• Adrian on devzone

Victor

• Get Google Analytics on the site
• Prepare OpenCV talk

Ethan

• .

From Last Time

Gary

• Send note to Vadim about creating a user wiki (with log in and password)
• See about forwarding old site to new site
• Cost out ECCV
• Send Victor an invite letter for GSoC Mentor summit. Date + week past

Vincent

• .

Ilya

• .

Vadim

• Update tutorials
  • Make sure we have installation on mac

Victor

• Talk to NVidia team about giving tutorial
• Cost out ECCV

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• schwag
• Hilton’s code
• 2.4.3
• ECCV
• speed priority
• OpenCV on Amazon
• repo for common stuff (logos)

Minutes

• Shishkov will update the Mac tutorial
• NVidia will fund own person for ECCV.
• talk at the university of ROME: scheduled
• tutorials ready for ECCV. Get USB keys done the day before
• schwag: contacted the online store, waiting for bug fix
• ECCV schwag: T-shirts a bit cheaper, stickers design ready
• Hilton’s code for training: ready to go, we need training.
• 2.4.3 delayed: tests are failing, bugs need to fix, hackathon at the beginning of October. Release November 1st.
• OpenCV.org: add logos/stickers in the downloads section
• Eric: Java. Beta module: deactivated for now. Needs more testing,really cool
• Maria and Ilya: actually joint work for training

Vadim

WR

Anatoly

WR

Ilya

WR

Action Items

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

From Last Time

Gary

• .

Vincent

• send Vadim Hilton’s repo
• put logos on website

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• ECCV tutorial
• FABMAP … other useful things from Willow such as features …
• Foundation
  • Intelligence Co
  • Audience
  • GSoC
• Website improvements
  • cheatsheet?
• Hilton’s code
• Transparency
• VO
  • Model capture
• Basic line fitting stuff for industrial. Use nlopt? Or duplicate that functionality
  • look up what is done.

Minutes

• ECCV tutorial
  • Tegra team are preparing for Android
  • Everyone meet the night before to burn USB keys etc
• FABMAP engine BoW image retrieval already in GitHub
  • in contrib.
  • 3rd party things will move to external modules
  • Extra is place holder for test data etc. Contains training sets — cascades for latent SVM
    • New cascades are in data/objcascades: Profile faces and silverware. Pretty good results for profile. Silverware works on only vertical, so need to rotate images
• Foundation
  • Tune OpenCV main algorithms for MIC, contests
  • Marketing budget … motivating use of algorithms/contests, tutorials
  • grants
• GSoC
  • Ilya is ready, Victor still needs visa …
• Website
  • Cheatsheet is in code.opencv.org resource section. Needs to be off of docs and opencv.org
• Hilton’s code, multi-part detector Deva Ramanan’s work
  • Detector in C++, trainer is in Matlab/Octave
    • need to move this along in separate module & Hilton improve along with thesis. Source and binary compatibility
• Transparent
  • Accurate models ~1mm. But need to improve segmentation. Not independent segmentation, but info from several images
  • Doesn’t depend on symmetry
  • We’ll discuss transparent when Ilya is here
• Model capture
  • RGBD odemetry for poses
  • Think about ways mesh and texture map
• line matching
  • Chamfer map — opencv sample/cpp/chamfer finds Apple logo
  • distance transform
• Andrea going to ECCV for tutorial. Need to meet the evening before

Vadim

[Bug Fixing]

over 15 tickets in OpenCV bug tracker have been closed.
[Tests]

Numerous OpenCV test failures (~10) on Windows, Mac, fresh Ubuntu versions have been fixed. They were

mostly caused by the incompatible libjpeg library. The plan is to move to .png files in OpenCV tests to avoid

such problems in the future.
[Optimization]

Evgeny added cross-platform threading to Cascade Detector (which is used for face detection, for example)

and HOG object detector.

Anatoly

WR

Ilya

Model capture of transparent objects [Ilya]

Improved the model capture of unpainted transparent objects. Developed a more robust algorithm

to remove artifacts on top of a model and implemented a 2-step coarse-to-fine approach to produce

models.
Collected a new dataset with 10 transparent objects to experiment with model capture.

Evaluated the algorithm using an object which has a shape of a truncated cone (see attachments

for an example of a train image and the computed model). This shape allows to produce the ground

truth model analytically and compare it with the created model. Measure of quality is partial directed

Hausdorff distance that is a percentile of distances between a point of the first model and the nearest

point of the second model. Results:

Two assumptions are made when reconstructing objects:
1. Object stands on a support plane.
2. Object is hollow and has open top (the fundamental limitation of the approach is inability to r
econstruct concavities).

Other:
Arranged travels to the GraphiCon conference to present our work on transparent objects and to the
USA to attend the GSoC summit and make a new demo on the PR2 robot at Willow Garage.

Model capture of opaque objects [Maria]
Complemented the graph structure to refine locations of object points too. We model each object point using
its 3D coordinates and minimize its distances to corresponding points from several frames. The current approach
gives slightly more accurate models than the models from the graph for the camera poses only. It seems the results
can be better if we consider the local structure of each model point too (e.g. as it’s done by Ruhnke).
See examples of reconstructed models in the attachments (Note: circles grids in the training frames are not used
for the odometry).

Action Items

Gary

• Contact Audience
• Alexander from Itseez, Andrea for Tegra coordinate with Vincent for loading USB keys
• Other: grants

Vincent

• What is the status of the new work on feature detectors descriptors.
• Ask Hilton to prepare multi-part detector as a module (docs, tests …)
  • Maybe he can have ongoing part in this as his thesis advances, his stuff is already out there

Ilya

• .

Vadim

• Send email CC’ing me, Vincent, Andrea and Alexander to coordinate meeting night before
• Mail to Alexander to be able to put a cheatsheet up and get yourself permissions for changing the site

Victor

• Case for foundation. Help with financials. Late Nov.

Ethan

• .

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Getting off lots of travel
• Other methods of funding
• Willow check
• Code status

Minutes

• OpenCV linux boxes … with service contracts
• Service contracts
  • Build bot
    • 1. Make sure build bot functions for a particular build /once a year (we just run build bot and builds that specific build. fails are your problem)
    • 2. Want to keep build to be consistent for each release of OpenCV. Build pass all tests. /year (builds are our problem with releases)
    • Call center
    • Movies and tutorials. Setting up environments
• Fitzg at MS
  • MS Accepted license, can use internally
• At GSoC collaborate across graphics, image types
• Willow Octo map grids, calibration — arm alignment, arm planning. Grasping
  • parts based detector
• Working on 2.4.3
  • End of this week probably
  • Make all the tests pass
  • Main highlights
    • GSoC python samples, denoising, new optical flow, new cascades for profile faces and silverware
    • lots of bug fixes.
    • Mobile support Android 4.1, iOS 6 (should get Windows mobile in)
    • parallelization, desktop Java, OpenCL (Chinese Academy of Science … but really plan this for 2.5, make transparent, if libraries are available, it will be used automagically).

Vadim

[OpenCV 2.4.3rc preparations]

12 more tickets have been closed; over 2 dozens of pull requests have been integrated

(most of the integration work done by Tegra team).
There are just a few things to do before 2.4.3rc is ready:
1. make our buildbot “green”. This is what we are currently working on. Most of the

problems are actually “false alarms”, but investigation of fixing some other problems

really unveiled problems in OpenCV OR the environment. For example, we found that

GCC 4.6 in Ubuntu 12.04 x86 is pretty buggy; when OpenCV is built with “-O3” option,

many tests fail on Ubuntu 12.04 x86 (and pass successfully on any other configuration).
2. make the actual packages for Unix/Mac, Windows, Android, iOS; test them; upload them to SF.
3. write the release notes (that’s my TODO); post the announcement.
[Automatic testing of pull requests: Evgeny]
Evgeny wrote a script (it’s beta for now and available only within Itseez) that allows

github.com/opencv/opencv admins to test each pull request using buildbot. In other words,

the patch from the specified pull request is applied to the specified branch, and this code is

compiled and tested in 3 configurations (Ubuntu, Windows, Mac). The next step is to make

buildbot to test each new pull request automatically and post specially formatted comment

to github. This way the process of pull request integration should be simplified a lot.

Anatoly

WR

Ilya

Model capture demo [Ilya]

Finished preparations to the Willow visit (got instructions from Maria how to use her model

capture code, collected with Maria a new dataset with a moving Kinect on a box to simulate PR2).

Attended the Google Summer of Code Mentor Summit.

Installed Maria’s model capture on a Willow machine. Collected a dataset for model capture

from a PR2 robot. However, the robot was controlled manually using RViz and it resulted in

too jagged movement. Plan to use the joystick for robot control and re-collect the data.
Model capture [Maria]

Captured with Ilya a dataset of a new type: rotate a Kinect around the table with an object

(our previous datasets had a fixed Kinect and the rotated table). Current model capture code

coped with such case of a moving camera too (odometry gave good poses and this is the base

step to get accurate models). So we hope that this’ll work on the robot. See the result in the

attachment (note: circle grids are not used).

Another iteration of the code refactoring was done.

Faced with the following problem on the new dataset: on stage of simultaneous refinement

of camera poses and model points we got several NaNs in Hessian of camera poses during second

iteration. I checked that we don’t pass NaN values to the graph and other hypotheses about the

reasons of NaNs, but without success yet. Unlike all our previous datasets (~25) where we didn’t

have this problem, the new one has bigger object, as result the graph has ~5 times more vertices

(~0.5M). Added the check of iterations with bad results to apply only success refinement.

Continue to investigate the reason of NaNs.
OpenCV [Maria]

Added 3 new tests on core module: cv::merge, cv::split, cv::phase.

Action Items

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Release 2.4.3
• New MSER
• DAFT functionality
• Atom
• Surface/Win phone
• Transparency
• Model capture

Minutes

• Release candidate 2.4.3 is out https://github.com/opencv/opencv/wiki/ChangeLog#243-rc
• Chinese Academy of Science working on OpenCL with …
• Mobile I-chips
  • Port, tune flags
  • Work interoperability: new sensing, perception data formats
  • Work on tutorial style examples
  • Work on real demo (image retrieval, summarization?)
  • Optimize based on above bottlenecks plus stitching, calibration, Features2D, Calib type stuff
• New MSER Matas. Evginy working on ER retrieval. Letters or structural shapes don’t correspond to MSER but ER.
  • Image has few millions of ER. Extract and analyze each region in constant time. Prune the uninteresting. Get CC of things that are similar to shapes that you are looking for
• RGBD module in OpenCV for simple convenience functionality. Alexander look at it.
  • Features 3D to encapsulate this data.
  • DAFT in WG. More general version in Ecto repository.
  • Need feedback on what should be there
• Synthetic view of an object. Visualizer for 3D data
• SFM module too, probably should be separate from RGBD
• Transparency
  • Collected data of bottles.
  • Opaque object model capture
• Model capture
  • Model capture going. Using ROS, publishing odometry online model learning
    • Robot captures model, spits out point clould
    • Vincent on auto training for linemod
    • Review on Thurs. 5 points. Same people … maybe one more.
      *

Vadim

[OpenCV 2.4.3rc]

The release candidate has been released. The changelog is here: https://github.com/opencv/opencv/wiki/ChangeLog#243-rc

The final 2.4.3 is expected on Nov 2nd. Big thanks to Tegra team who helped with the release (as they do regularly during this year)!
[Development process]

Evgeny continues work on the technical aspects of the new workflow. It will be a page (it’s now in alpha version) that will allow

developers to see the status of their patches. Perhaps, commits from regular experienced contributors will not require the use o

f too heavy pull request mechanism.

Anatoly

WR

Ilya

Model capture demo [Ilya]

Wrote a demo application for a robot which grabs data from a RGBD sensor and constructs a model using Maria’s algorithm.

Visualized odometry poses in rviz. A robot path is tracked with Maria’s odometry and it is visualized in rviz simultaneously (see a screenshot).

Experimented with model capture on different robots and objects. Results are good and quite robust (attached several examples).

Thanks to Maria for help with the demo.
Other:

Collected several datasets of transparent objects at IPI.
Model capture [Maria]

Experimented on the dataset from a robot to understand the differences with our datasets that can be considered in the algorithms.

Implemented online frame-to-frame processing.

Implemented refinement of no-keyframes poses within segments of already globally refined trajectory of keyframes.

The goal is to fill missing and invisible information in keyframes set and to get more accurate filter of outliers in the model

point cloud. First result of no-keyframe poses refinement is unsatisfactory. Investigating the reason of this now.

Action Items

Gary

• .

Vincent

• Set up RGBD module feedback on list
• Set up SFM module feedback on list.

Ilya

• Make transparent bottle data available to me

Vadim

• .

Victor

• .

Ethan

• .

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Fun with funding
• 2.5
• Background subtraction

Minutes

• 3 people allocated to new port
  • Many of the optimizations should be applicable
  • Need to set up SOWs and normalize the levels
• 2.4.3 is out
  • This is pushed out to ROS build farm, no problems found!
• 2.5
  • SOW this week
    • Transition to micro modules
    • Other people can more easily add modules and share
      • Makes the whole thing a distributed library like Boost
    • Better support for higher level languages (Matlab, Octave)
  • Long list. Share by End of Week
  • Separate C API
  • Future proof new interface
  • Khronos compliant
• Transparent data, how to prepare. Discuss with Ilya later
• New background subtraction code. Looks good
• Victor is scheduling his visit week of 12. Khronos visit.!! Will coordinate time and G meeting.

Vadim

[OpenCV 2.4.3]

The final OpenCV 2.4.3 is out. The announcement is put to http://opencv.org. After the release candidate we have

fixed in total ~10 bugs together with the Tegra team (mostly various compile problems) and made our buildbot

(build.opencv.org) green. Well, sometimes it gets orange or red because of some internal instabilities, but in general

we fixed most of the test failures by 2.4.3. Currently, we are planning 2.5; the draft version of the proposal should be

ready this week.

Anatoly

WR

Ilya

Demo of model capture

Collected a new dataset from a PR2 robot for subsequent experiments with model capture at Itseez. The dataset contains

33 bag files; the robot makes one loop around a table in each bag file.

Conducted more experiments with the model capture. Attached several examples of new results (the bear was scanned

while lying on its back). However, these models don’t have points for regions which a robot didn’t see. For example Mini-mouse’s

chin is missed.

Vincent and me finished the preparation of the demo and showed it at Willow Garage. RGBD odometry, model capture and

LINE-MOD were demonstrated as Vincent already wrote. The demo went well in overall, however, tracking was not robust and

loop wasn’t closed because depth was corrupted by skylight as we checked after the demo. So we had to use a precomputed

model which we created in completely the same setup as the demo but at the night before. Still we showed several good

models produced with a robot so people could see and appreciate results of the algorithm. Also other parts of the demo

went without such problems.

Thanks a lot to Vincent and Maria for crucial help with the demo.
Model capture of opaque objects

Found the reason why my last week refinement of no-keyframe poses gave bad results. Checked several approaches to

do this refinement. The final version is rebasing no-keyframes within globally refined keyframes and optimization of camera

poses graph for each keyframes segment of the trajectory. This approach is very fast and gives accurate results (see the attachment:

green point clouds are all globally refined keyframes, other color point clouds are all no-keyframes). The other checked versions

were more complicated (with ICP and RGBD edges), but did not show improvement and they worked unacceptably long time

(even by segments).

Implementation of view angle histograms of model points (for more stable outliers filtering; using all frames) is in progress.
Other:

Monday was a national holiday in Russia.

An extended abstract about our work on transparent objects was written for a Summer School on “Robot Vision and Applications”.

Action Items

Gary

• .

Vincent

• Funding

Ilya

• Scripts or how to prepare the transparent bottle data.

Vadim

• Send out 2.5 specs

Victor

• .

Ethan

• .

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• New funding
• I funding
• 2.5
• CLM

Minutes

• Android port works perfect for Vincent on first try. Kudos to the Android team!
• OpenCV Manager will select what’s needed, pull in the correct version na
• The new descriptor is performing extremely well, beating all prior
  • logpolar trick for working with scales
• Maybe we need to get 2.4.3.1 fix for 3×3 inversion for signal precision breaks down with illcondition. This was turned on by SSE
  optimization but has to be taken out.

Vadim

[post OpenCV 2.4.3: Bug fixes]

Several bugs, found in 2.4.3 have been fixed: bad accuracy in 3×3 matrix inversion, broken update method in

LBPH face detector, data races in parallel video reader/writer, broken build on Windows with OpenGL enabled …
[Infrastructure]

Evgeny continues to improve the automated OpenCV build and testing infrastructure:

- integrated developer branches testing functionality into web GUI;

- improved the distributed testing of the submitted pull requests.

- fixed several try jobs merging into a single build;

- modfied other build definitions: e.g. no “success” if builds with warnings;

- “merge button” proof of concept prepared. The idea is that the pull requests that passed all the tests and got approved could be

automatically merged into the repository.
[OpenCV 2.5: the Proposal Draft is ready and has been discussed]

Here is the document that I prepared: https://docs.google.com/a/itseez.com/document/d/1L4pCxMXLNyjFvZAbH-xDZghQCKDhbckjBAYFssPHWD8/edit

It has been briefly discussed with several developers from various teams. OpenCL team is also in the process of reviewing it.
[The proposed changes in OpenCV roadmap: 2.4.4 and 2.5.0]

Since there is a lot planned for OpenCV 2.5, it would be very challenging to prepare some alpha release by February. So instead the

suggestion is to:

release 2.4.4 in February

release 2.5.0 in May

Anatoly

WR

Ilya

Ilya is on vacation from Nov 7 till Nov 16.

Maria is on vacation from Nov 7 till Nov 20.
Other:

Ilya won a $850 travel grant to attend “Robot Vision and Applications” Summer School. Thanks to Vincent for writing a recommendation.

Action Items

Gary

• Set up funding allocation meeting with Victor/Vincent
• Ping I on getting new project going
• Get creative going under ubuntu
• Read & comment on OpenCV 2.5 plan

Vincent

• Read & comment on OpenCV 2.5 plan
• Surface noise level of create

Ilya

• Read & comment on OpenCV 2.5 plan

Vadim

• Read & comment on OpenCV 2.5 plan

Victor

• Read & comment on OpenCV 2.5 plan

Ethan

• .

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• C++ exceptions
• blog items
• wikipedia
• 2.5
  • Matlab/Octave interface

Minutes

• Evgeny did cool work so that we can see, via the build bot whether pull requests work before we pull them
• Google hates exceptions. Apparently there is no gcc pluggin that handles exceptions that is not GPL (although, note that Linux kernel is GPL).
  • But GPL free exception support in C++ is coming. Maybe leave for now
• Send blog items to Alexander Shishkov
• Updated links around the web, need to get someone on this full time
• Matlab/Octave interface — perfect for next Google summer of code
• Put comments into 2.5 document prior to publicizing it

Vadim

[Infrastructure]

Some more results from Evgeny who works on OpenCV infrastructure:

- finally set tdr parameters to tegra-win32 buildslave eliminating timeout-related test failures;

- successfully carried out more tests of “merge button”;

- refetched commit history at code.opencv.org to avoid any whitspaces in ‘Repository’ section;
The latest version our the integration page, where the status for each pull request is attached.
[Edge-aware demosaicing]

Ilya Lavrenov has implemented edge-aware demosaicing algorithm. It has been threaded with nearly

linear speedup on dual-core CPUs. The regression tests are there too.

• New system to see if pull requests work via the build bot before pulling

Anatoly

WR

Ilya

Maria will return from vacation on Nov 21.
Ilya returned from vacation on Nov 20. Started to write an extended version of our GraphiCon’2012

paper for post-proceedings which will be published in Springer Lecture Notes in Computer Science.

The new paper will contain more details about our previous work on transparent objects using a stereo

camera.

Action Items

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• Ask Alexander Shishkov about putting in blog items
• Ask Alexander to investigate whether Google ads would be worth it. Buy servers, infrastructure

Victor

• .

Ethan

• .

From Last Time

Gary

• Set up funding allocation meeting with Victor/Vincent
• Ping I on getting new project going
• Get creative going under ubuntu
• Read & comment on OpenCV 2.5 plan

Vincent

• Read & comment on OpenCV 2.5 plan
• Surface noise level of create

Ilya

• Read & comment on OpenCV 2.5 plan

Vadim

• Read & comment on OpenCV 2.5 plan

Victor

• Read & comment on OpenCV 2.5 plan

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• Ads?
• New organization
• Update on new client
• Content plans

Minutes

• Sorry. Meeting pre-empted by OpenNI discussion and by VIP visit at Industrial Perception.
• 2.5 Module division proposal below

Vadim

[Infrastructure: Evgeny]

Evgeny has practically finished the work on the pull request testing page:

- “merge button” finally can be used, several pull requests were intgerated;

- fixed merge button behavior in case of any git command failure;

- published pull requests testing page at http://pullrequest.opencv.org, so it is now visible for everyone

(thanks to Alexander Tsyplenkov and Alexander Shishkov)
[Demosaicing: Ilya Lavrenov]

Ilya improved implementation of edge-aware demosaicing algorithm, now the performance should be much better.

bayer2RGBA has been parallelized. The parallel version works fine on Mac/Win and now is being debugged on Linux.
[Bug fixing, Misc]

several pull requests from users are at the negotiation stage (improved capturing from Prosillica cameras, all-in-one

connected component extraction algorithm, …); documentation build problems have been fixed; 3×3 matrix inversion has been fixed
[OpenCV 2.5 discussion]

The rough new subdivision of OpenCV has been proposed:

https://docs.google.com/a/itseez.com/spreadsheet/ccc?key=0AoYFDB08hRs6dHFzMzRtVFppdXFtMzQzQ1JxRmZXd1E#gid=0
it will likely be refined.
[Infrastructure: Build server]

it appears that we may need the build server, but not as urgently as we initially thought, we will try to optimize the amount of

work that the current buildbot does. Terribly sorry for alarming you. We will get back to this question next year.

Anatoly

WR

Ilya

Transparent objects [Ilya]

Finished the paper on transparent objects for the GraphiCon’2012 post-proceedings. Wrote about

our algorithm of edge model creation inspired by bio-informatics and added an example of working

with textureless objects.

Prepared a poster about transparent objects for the IEEE RAS Summer School on

“Robot Vision and Applications”.

Sifted through ECCV 2012 papers. Found one paper on the same problem that I was struggling

for a month in model capture of transparent objects: “N-tuple Color Segmentation for Multi-View

Silhouette Extraction” by Abdelaziz Djelouah et al. Unfortunately, It cannot be applied directly

because they wrote “Failure cases have been observed with configurations where the background and

foreground color distributions are similar”. Going to study it and linked papers more thoroughly

to check for ideas that can be useful for transparent objects too.
Other:

Gave a talk on Projective Geometry in Computer Vision at a student school.
Model capture [Maria]

Returned from vacation on Nov 21. Refactoring the code and reading more papers about model

capture (stable camera tracking, in-had scanning, etc). Both are not finished yet.

Action Items

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• Ask Alexander Shishkov about putting in blog items
• Ask Alexander to investigate whether Google ads would be worth it. Buy servers, infrastructure

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• OpenCV hiring needs
• 2.5
• Donors

Minutes

• People asking how donation is used
  • We have to make this clear. Soon moving to larger support by Willow, Intel and Google, but other’s asking. So far we have
  • http://opencv.org/contribute.html
• Noise model for depth for rgbd
  • sensor module declare sensor type with noise model.
• 2.5 have early prototype. But 2.5 is intended to be stable for several years
  • By next week get shell to compile
  • Pospone book for this. API agreed by API
    • Set of modules have been changed
    • Some functionality moved to different modules
    • C API moved to (new) compatibility module
      • Legacy module for older stuff
    • Style of using class instances, you use smart pointer
      • Like NVidia capture method
    • Look at code samples and start changing

• Could use a maintainer
  

  
  
  The maintainer’s OpenCV time is spent on three activities.

- Communication (60%)

Mailing list discussions on general design, fielding user
questions, diagnosing bug reports; reviewing, commenting on,
suggesting alternatives to, and rejecting patches.

- Integration (30%)

Applying new patches from the contributors while spotting and
correcting minor mistakes, shuffling the integration and
testing branches, pushing the results out, cutting the
releases, and making announcements.

- Own development (10%)

Scratching my own itch and sending proposed patch series out.

Vadim

[OpenCV 2.4.3.1]

Very minor update to 2.4.3 has been prepared (by Tegra team) and is available from

https://github.com/opencv/opencv/tags. It’s so minor that we will not make any public announce.

Most of the fixes relate to Android.
[Infrastructure: Evgeny]

Some further improvements have been done to the pull request testing page (the “external”

variant of the page is available at http://pullrequest.opencv.org):

- “merge” button gets different colors, depending on the comments and build status

(e.g. when the last comment is not :+1: or :shipit:, the button gets orange rather than green)

- the last comment is printed at the status page

- the behavior is now more robust: we set time limit for each step of precommit build,

thus invalid code does not break the buildbot.
[Demosaicing: Ilya Lavrenov]

Ilya fixed the remaining bugs in the parallel edge-aware demosaicing and added parallel

version of bayer2Gray.
[OpenCV 2.5 concept]

I have started preparing the concept version of OpenCV 2.5, which is now at

https://github.com/vpisarev/opencv/tree/2.5_concept (completely broken for now).

The compile problems will likely be fixed by the end of the week; the tests will take

some more and then we will continue to refine it together.

Anatoly

WR

Ilya

WR

Action Items

Gary

• Get accounting info to Vincent
• Contact Adrian about book pow-wow

Vincent

• Writing down donation pages

Ilya

• .

Vadim

• Send schedule so we can sync book with 2.5
• Send plan also for ML module

Victor

• .

Ethan

• .

From Last Time

Gary

• .

Vincent

• .

Ilya

• .

Vadim

• .

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• 2.5
  • Book info on 2.5
  • ML info
• Donation page
• Website
• Intel
• transparency

Minutes

• Release 2.5
  • Studying partitions and refactoring. Illiminate lessor used stuff. Slim everything down
  • Big blocks such as ML, how to handle? Other things are much easier
  • Plan for 2.5 to compile in Jan
  • How to avoid cycles in dependencies for things that can be members of several categories?
  • Module size … most quite small, some
  • Compatibillity module to help with transition
    • Beta end of April.
    • Release is planned for end of June
• Sync 2.5 with the book
  • Telcon Vadim with Adrian and Gary
  • Look at github https://github.com/vpisarev/opencv/tree/2.5_concept
• Donation page
  • Vincent
• Website, need instructions on how to modify for all admins
  • News
  • Blog needs to be set up. Separate site and reference it
  • Donation give/get needs to be set up
• Evangalizing position will come up in 2013.
  • Need to find someone who can step in full time and
    • drive code bounties,
    • website,
    • help with fund raising
• Vincent talk at UCSD
  • Vincent to France Wed.
  • Mid Feb. Set up talks (Pr Nk)
• 2.5 content
  • AR big (duh)
  • Model capture
  • precise tracking
  • SLAM
  • Movie. Aggarwal Panorama
  • Motion capture
• Transparent object
  • New (secret) approach is working well and fast
  • Combine with existing techniques, use above to initialize
• Model capture
  • Work with better trajectories.
    • Now we circle objects and reconstruct with path completion
    • Random trajectory next
      • We need ease of use
    • Colored points
      • linemod to work in training with this
      • Meshlab to mesh
• Vadim on vacation next week to end of December.
• Register OpenCV coverity http://scan.coverity.com/start/

Vadim

[Infrastructure: Evgeny]

- We significantly reduced the load of our buildbot, in order to make it more responsive and free some resources for more

extensive testing. So far, the number of builders have been reduced from 50+ to ~30.

- All the invocations of the buildbot builders are now recorded in a DB, so we have more robust and much more

responsive http://pullrequest.opencv.org.

- A separate project, called meta-opencv, is now created at code.opencv.org. We will use it as a place to work on

OpenCV infrastructure. In particular, we put there our buildbot scripts.
[Demosaicing: Ilya Lavrenov]

Ilya started studying SSE2 intrinsics and as a first assignment he optimized the edge-aware demosaicing algorithm.

The SSE2-optimized version is ~4x faster than C version. Now he is working on optimizing some cases of the resize function.
[OpenCV 2.5 concept]

Vadim continues to work on the OpenCV 2.5 concept at https://github.com/vpisarev/opencv/tree/2.5_concept. So far the

modules hal (it’s almost empty for now), core (except for XML&YAML I/O), imgproc (except for contours and few more functions),

bgfg (100%), optflow (50%) are ready. It can take some more weeks to make it completely ready. Given the coming holidatys,

I expect the prototype to be ready and pass some tests to be ready by the early January.
[Plans before NY]

Evgeny plans to switch to non-intensive maintainance mode of the OpenCV infrastructure and spend more time on the actual

code work, e.g. Matas approach for fast connected component analysis.

Vadim plans to take vacation from the second half of December.

Anatoly

WR

Ilya

Model capture of opaque objects [Maria]

Finished next step of the code refactoring.

Implemented missing functionality: getting a model, reading from / writing to a ply file.

Prepared and gave a talk about model capture at Itseez.

Implemented refinement of an object mask after table segmentation. Sometimes outliers in this mask were

a reason of dangling points in the final model.

Implemented the old idea to compute histograms of view angles for each point using all frames (including

refined no-keyframes) and filter the points basing on such histograms. But it appeared that this filter is not

more accurate on our data than current simple filter by correspondences count.

Made simple experiment to improve a model texture: implemented texturing version that set for 3d point a

color that is weighted sum of its projections color on several frames (the weight depends on point normal).

Again this does not give better texture visually.
Plans:

Read more SLAM papers to go to scanning with arbitrary camera trajectory.
Model capture of transparent objects [Ilya]

Studied papers “N-tuple Color Segmentation for Multi-View Silhouette Extraction” and “Automatic 3D object

segmentation in multiple views using volumetric graph-cuts” in more details. The last paper is very similar to

our current approach. However, both papers don’t solve our critical issues and shouldn’t work well in our case

when background and foreground are extremely similar.

Revised our problem of 3D reconstruction and formulated it more rigorously. It is energy minimization on a

factor graph. The issue is the graph has loops and factors have order of 10-200 terms (it depends on the

algorithm parameters). I don’t know how to find the global minimum for such a graph. Going to try to devise

an adhoc global optimization scheme by exploiting the specific structure of the graph, otherwise use

approximate inference or re-formulate the problem.
Other:

Ilya was on vacation and he attended IEEE RAS Summer School on “Robot Vision and Applications” in

Santiago, Chile (December 3-7).

Action Items

Gary

• Send Vincent balance pdf
• Send Vadim Intel SOW

Vincent

• Donation page
• Set up talks for when you come back in Feb.
• Check out http://scan.coverity.com/start/
• Ping Celia, make sure Willow has OpenCV wire info

Ilya

• See about resources for doing vision work for use & publication
  • Transparent stuff
  • Linemod features that train well from model and work well with real camera test

Vadim

• Write Adrian and Gary
• Decide on book

Victor

• .

Ethan

• .

From Last Time

Gary

• Get accounting info to Vincent
• -Contact Adrian about book pow-wow-

Vincent

• Writing down donation pages

Ilya

• .

Vadim

• Send schedule so we can sync book with 2.5
• Send plan also for ML module

Victor

• .

Ethan

• .

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Agenda

• 2.5
• Text localization
• Industrial
• Barcode
• Connected components general framework? (Jason Newton code)
• Book considerations
  • ML
    • good examples
  • Code examples
• Transparent
• Model capture
• Vacations: This is the last phone meeting of the year. We will convene again Jan 8th

Minutes

• 2.5
  • Some work. Vadim dealt with pull requests mostly
• Text localization
  • But good technique for connected components/extremal regions where they can be split
    • Different channels, H, S, V. Different thresholds on each channel. Raising threshold, watch the extremal region emerge. Use classifier to reject non-letter components
    • Constant time algorithm … so optimized this is fast constant time
  • Would like to make general — the training step. For now, just get it working, then optimized
• We have a new fast connected component contributed by Jason Newton. Cool!
  • Each algorithm, even when it is “finished,” takes a month to get ready for real inclusion.
    • This is necessary, we no longer want to bloat up on junk. Everything should be engineered, have a reason to exist, work well in the entire framework.
• Industrial
  • Should there be a module in this area?
    • Probably yes, but would be kept minimal. Lines, templates, barcode aides (not worth doing too much in barcodes, this has already been solved)
    • Barcode normalization, subpixel lines
  • question of priorities
  • Probably good for longer term interns. Probably too complex for summer of code kind of effort.
  • Crowd source funding?
• ML
  • Letter recognition
    • 6 different algorithms NN, random trees, boosting, svm, bayesian, knn
    • NN: Boolean vector of 26 elements
    • Boosting: Unroll data into a vector. Add categorical variable as last component of the input vector (binary classifier)
    • Random trees: Categorical classifier that gives the letter index
    • K-NN: Simple to use, several nearest features
  • Python for hand written digits. Training and testing (svm, knn)
    • Image with various digits, trained and recognized by sliding windows
• Transparent
  • transparent silhouette in rotation for symetric objects
  • Gary’s technique for recognition and pose. Worth investigating more/paper
• Model capture continues to improve. Getting accurate texture is hardest part
  • Collecting data, hand held
  • Examining SLAM techniques for better tracking/loop closure
  • Better sensors … would yield better results obviously. Motion blur a problem, global shutter a solution. Kurt may solve, working on it.

• Vacations: Gary gone Saturday back 7th; Vadim back 7th Jan; Ilya back 7th.

Vadim

[New functionality]

- Evgeny is now working on implementing “Real-time scene text localization and

recognition” by L. Neumann and J. Matas. The draft non-optimized implementation

of the first most expensive preprocessing stage is ready; the next steps will be:

extensive optimization of the first phase, adding classification and the second stage.
- Jason Newton contributed fast algorithm for connected component extraction, which is

about to be integrated to the master branch (once we eliminate test failure on MacOSX).
[Optimization]

- Ilya implemented SSE2 version of resize area fast. Here are the execution times

in ms to process FullHD image on Core i5 2.5 GHz:
Resize Area Fast     CV_8UC1     CV_8UC3      CV_16UC1   CV_16UC4

C version             502         1292          380         1982

SSE2 version          112          510          189         1218

Speed up              4.5          2.5           2          1.6
Now Ilya is studying HOG algorithm and will accelerate it further.
[OpenCV 2.5 concept]

Vadim did some more work on OpenCV 2.5 concept. Some more functions in imgproc have

been rewritten in C++, optflow module now compiles; the work on geometry module has been started.

Anatoly

WR

Vincent

on my TODO:

- not done yet with the foundation/donation page

- talk with YYYY in progress. XXXXX too

- we are now part of coverity ! My intentions were twofold:

- find new bugs in OpenCV: coverity is a static code analysis (the best out there) and they provide free analysis for Open Source software:

http://scan.coverity.com/all-projects.html

(we should be on there soon)

We did find some bugs (e.g. some memory leaks) but only under certain conditions or in C code/rare API.

Still, they should be fixed at some point. I sent a list to Vadim about the leaks.

- being on coverity is also a way to show that we care about industrial quality. Linux, Python, OpenOffice, PHP … are on coverity.

1100 industrial companies pay for it : Adobe, ARM, EA games, EADS, LG, Lenovo ….

I still need to setup automatic uploads of a build once a week. Should be trivial, I just need internet ….

- Switchover on funding for the new year is on track

Ilya

Transparent objects [Ilya]

Discussed with Victor our problem of energy minimization on a factor graph

in reconstruction of transparent objects. Decided to not use the graphic models

framework because  we have high-order factors in our graph and it is a too difficult

problem for standard algorithms. Started to think about an alternative approach.

Supported Willow Garage in usage of our transparent objects pipeline. Experimented

with training and detection on their data. Training on unpainted transparent objects

didn’t work because it is still in development so a simple semi-automatic training was

implemented as a workaround. Detection worked ok, video with results was prepared and sent  out.
Plans:

Develop reconstruction of unpainted transparent objects which will be less generic but

more robust (it will use the rotational symmetry assumption).
Model capture [Maria]

Read a lot of papers about SLAM to replace current one loop closure by an arbitrary graph structure

from a handheld sensor. Still there is no final decision which algorithm we should implement but we want

to have more robust odometry, false poses filter and a loop detector.

Switched to use Xtion Pro Live instead of Kinect because for the current odometry we need high FPS of

synchronized rgb and depth from a handheld sensor. Implemented a sample to capture the data from Xtion in

the base format we use for the model capture now.

Captured several datasets from the handheld device. The capture conditions of the first object were close

to ones we satisfied before (on a robot): one loop, relatively slow camera movement. But this time there was

camera jiggle from me and movements of my hands. The object was reconstructed successfully (see the attachment).

The capture conditions of other datasets were more extreme (fast movement, etc.). In these cases sometimes

trajectory was broken. Analyzing the problems of the odometry and how we can increase its robustness in more

extreme capturing.