Photogrammetry System for Small Scale Objects
Collection of background information.
Goal is to create a system that allows accurate and reliable 3D scanning of small objects, both for use in computer applications (games, content, etc), and (functional) CAD.
Structured Light Scanners
We tried MakeItZone’s structured light scanners (Kinect, Ocipital Scanner) with Skannect. Discovered they’re affected by ambient light. Also only good for items ~50cm scale or larger.
Did some quick experiments with taking pictures with a cell phone and using meshroom for photogrammetry. Happily surprised to see results with small (~2cm) objects. Lots of noise, but can see the potential.
- understanding meshroom
- figuring out appropriate lighting
- discovering what reference info is needed (plain background + turntable, moving “floor” on turntable, moving around the object…)
- trying turntables and backdrops
- design and build a computer controlled turntable
- design and build an arm that can raise a camera in a circular arc, that keeps the camera equi-distant and always pointing towards the center of the turn-table
- application to generate turntable/platform pattern sheet with embedded ArUco fiducials
- automated image processing
- good initial experiments with DigiKam’s batch processing
- increasing sharpness/resolution; decreasing noise
- multi-frame super resolution
- NOT single image, or any “GAN” based algorithm
- these use techniques (neural nets) to create/“imagine” higher res details based on pieces of other images that could fit what would create the real blurred pixels.
- OpenCV has a superres module
- put (web) cam on platform and take multiple, slightly shifted images (and multiple exposures in the same spot?)
- multiple exposures in same place + averaging -> removing noise
- multiple images slightly displaced -> sharper details (there are limits- the sensor pixel size is fixed, and there’s only so much that can be implied by how things appear as they shift across pixels.)
- Guess: images need to be displaced in a plane orthogonal to centre of lens. If camera is rotated, angles to items, occlusions, etc will be changed by larger amounts and there will be more distortion.
- depth of field
- want (relatively) wide depth of field
- in lens: small aperture -> and long exposure
- computationally: macro shots + focus stacking- https://patdavid.net/2013/01/focus-stacking-macro-photos-enfuse.html
- color matching/grading
- white balance
- color matching (eg include reference color swatch in all images)
- background removal (
ffmpeghas chroma-key abilities that work with images)
- camera calibration
- creating pre-calculated camera location and pose info for meshroom
- Augmented reality
- Far future: using real-time point clouds from ARCore/ArKit as seed to meshroom
- ARKit augmented image capture app:
- initialize: mark side and top location of a rectangular prism that encloses the target
- calculate the rings of “snap” points (eg 10˚ spacing) where we want images captured
- run ARScene, showing transparent prism over the item to be captured
- indicate where the points are in 3Space and guide the user to move phone camera to the right location, and with the right rotation
- automatically grab images when camera is at the “snap” points. Grab multiple exposures if possible, for noise reduction, and superres.