“Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentation from Qualcomm
•
0 likes•508 views
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2022/06/tools-for-creating-next-gen-computer-vision-apps-on-snapdragon-a-presentation-from-qualcomm/ Judd Heape, Vice President of Product Management for Camera, Computer Vision and Video Technology at Qualcomm, presents the “Tools for Creating Next-Gen Computer Vision Apps on Snapdragon” tutorial at the May 2022 Embedded Vision Summit. The Snapdragon Mobile Platform powers the world’s best smartphones, XR headsets, PCs, wearables, cars and IoT products. Thanks to Snapdragon, these products feature powerful computer vision technologies that you can tap into to build next-gen apps. Inside Snapdragon is a hardware engine dedicated to computer vision–the Engine for Visual Analytics (EVA). EVA hardware acceleration gives developers access to high-performance, low-power computer vision functions to enhance apps that rely on advanced camera or video processing. The EVA includes a motion processing unit, a feature descriptor unit, a depth estimation unit, a geometric correction unit and an object detection unit. These blocks power high-level functions such as electronic image stabilization, multi-frame HDR, face detection and real-time bokeh. In this presentation, Heape does a deep-dive into EVA’s Software Developer Kit (SDK) and available APIs, such as Optical Flow and Depth from Stereo, and explores how these features can be integrated into your apps.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to “Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentation from Qualcomm
Similar to “Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentation from Qualcomm (20)
More from Edge AI and Vision Alliance
More from Edge AI and Vision Alliance (20)
Recently uploaded
Recently uploaded (20)
“Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentation from Qualcomm
- 1. Tools for Creating Next-Gen Computer Vision Apps on Snapdragon Judd Heape VP Product Management for Camera, Computer Vision and Video Technology
- 2. Computer Vision in Snapdragon Three function levels to provide comprehensive CV solutions 2 CV hardware Acceleration blocks to support and enable hardware, software and system designs in Snapdragon platforms 1 CV Algorithms to demonstrate complete workflows that provide state-of-the-art solutions to certain perception problems 2 CV end-to-end Applications in mobile, XR, Automotive and IOT market segments to enable unique and enhanced user experiences 3 © 2022 Qualcomm
- 3. Engine for Visual Analytics (EVA): Computer Vision Hardware Blocks 3 EVA1.x EVA2.x EVA3.x Object/Face Detection Mobile Camera / Video XR Auto IoT Optical Flow Depth Estimation Feature Extraction Geometry Correction XR & 3DR Object Detection HOG/SVM Feature – video encode with 30% BR reduction HCD/NCC Semi-dense OF GMO for video encode DFS 1080p@30 Video Bokeh Lens Distortion Correction ACF/RDF Face Detection Dense OF (SGM based) Dense motion map for multi-frame processing, sensor alignment DFS (SGM based) • Bokeh better quality • Visual special effect • XR 3D reconstruction HCD/ORB – Centralized ME for camera Flow improvement for XR 6DoF/VIO Exposure Compensation Motion and depth map warping LSR (in EVAa 3.5) XRA - DoH,DoG FREAK, R-BRIEF (In EVAa/EVAv 3.5) © 2022 Qualcomm
- 4. Optical Flow 4 Semi-Dense OF Dense OF Motion Density Every 2x2 block Every pixel Motion Accuracy 1/8 pixel 1/16 pixel Motion Range (X,Y) ±128, ±64 ±64, ±32 Max Resolution 1920x1080 1152x648 Confidence Map 8-bit 8-bit Frames per Second 60 60 Sparse Motion • Feature Point Detection, Local and Global Motion • Various Detector and Descriptors (Harris, DoH, DoG, FREAK) Dense Motion • Semi-dense Optical Flow (sDOF) • Dense Optical Flow (DOF) • Hybrid Deep Learning based Motion + OF Segmentation Enhanced © 2022 Qualcomm
- 5. Depth from Stereo Estimation 5 Depth from Stereo (DFS) • Super-pixel Segmentation on SLIC • Feature Extraction and Matching • Confidence Map and Post Processing DFS Engine Depth Density Every pixel Disparity Accuracy 1/16 pixel Disparity Level [0,63] Max Resolution 720P@60FPS Input Images Flat Area Detection Depth Map SLIC Map Confidence Measure Post Processing SLIC (Simple Linear Iterative Clustering) Census Feature SGM © 2022 Qualcomm
- 6. Geometric Correction Engine (GCE) 6 Low-power High-quality Warping • ICA maps output pixels to input pixels Effective Transformation • Sparse grid transformation (35x27 or 67x51) • Dense grid transformation ( 8 pixel grid ) • Perspective transformation (3x3 transform) Output domain Upscale + Offset Input/IFE domain Virtual domain Effective transform Virtual domain Offset + Downscale Effective Transform Sparse Grid Transformation Dense Grid Transformation Perspective Transformation GCE Use Cases • Lens distortion correction • Motion vector grid composition • Rectification Rectification Lens Distortion Correction © 2022 Qualcomm
- 7. Normalized Cross Correlation 7 NCC Supports Two Modes • Patch to Frame Mode • Frame to Frame Mode 18 8 8x8 templates: Prepared by application, can come from different sources Reference frame Patch-to-frame mode Reference frame Current frame Templates: All in the same frame Frame-to-frame mode Frame Matching Using Harris Corners and NCC © 2022 Qualcomm
- 8. Face Detection 8 Deep Learning based Face Detection (FD) • Min Face Size: 32x32 • Detection Accuracy: 95% • 1080p@60FPS • Multiple cameras supported Under Non-Ideal Conditions • Strong Backlight • Full Profile • Occlusions – Face Masks, Hats, Glasses, Sunglasses Strong Backlight Full Profile Occlusions © 2022 Qualcomm
- 9. EVA Architecture and Access 9 • The EVA APIs are exposed both from the CPU and Hexagon Processor sides • It includes both synchronous APIs and asynchronous APIs • There are direct interrupts between the Hexagon Processor and EVA cores for low latency communication • EVA includes embedded CPU primarily for task scheduling and hardware pipes • EVA hardware pipes are shared between certain functions Hardware Pipes EVA CPU Data API & Control Hexagon Processor EVA API CV App CV Engine CV Engine EVA Driver EVA Driver Firmware CPU OF/DFS GCE HCD, NCC, ORB, DS DDR © 2022 Qualcomm
- 10. EVA Feature APIs 10 EVA3.0 Features EVA API Image Warping evaWarp_Sync / evaWarp_Async Depth from Stereo (DFS) evaDfs_Sync / evaDfs_Async Normalized Cross Correlation (NCC) evaNccFrame_Sync / evaNccFrame_Async Optical Flow (OF) evaOF_Sync / evaOF_Async Feature Extraction (HCD) evaFeaturePoint_Sync / evaFeaturePoint_Async Feature Descriptor Calc & Matching evaDcm_Sync / evaDcm_Async Downscaler evaScaledown_Sync / evaScaledown_Async Pyramid Image evaPyramidImage_Sync / evaPyramidImage_Async © 2022 Qualcomm
- 11. EVA SDK Simulator 11 © 2022 Qualcomm
- 12. CV Use Case 1 Depth Map from Stereo Cameras (DFS) 12 Applications • Accurate Camera/Video Bokeh effect • Background replacement in video recording or Zoom call • AR/VR (3D Reconstruction, Video Passthrough, Occlusion) © 2022 Qualcomm
- 13. CV Use Case 2 Real Time Bokeh Effect using Depth Map from Stereo Cameras (DFS) 13 Applications • Accurate Camera/Video Bokeh effect © 2022 Qualcomm
- 14. CV Use Case 3 Dense Motion Map (DMM) for Video MCTF 14 Key Benefits of EVA • Register multiple frames with local motion compensated • Remove ghosting artifacts in combined video frames © 2022 Qualcomm
- 15. CV Use Case 4 Dense Motion Map (DMM) for Video MFHDR 15 Key Benefits of EVA • Estimating and compensating for motion is key to achieve high quality HDR video • Remove ghosting artifacts in combined video frames • Running global motion and local motion estimation simultaneously requires large amount of computation power © 2022 Qualcomm
- 16. CV Use Case 5 Face Detection (FD) and Face Landmark Detection (FLD) 16 Applications • Gender/Expression/ Emotion/Gaze detection • Avatar animation • Geometric personalization Qualcomm Deep Learning-based 3D face landmark detection reaches high accuracy in locating 115or 300facial landmarks © 2022 Qualcomm
- 17. Start Developing on Snapdragon 17 Capture at higher FPS Extend battery life Tap into hardware-accelerated CV features with an SDK not previously available © 2022 Qualcomm
- 18. Start Developing on Snapdragon 18 Xin Zhong Director, Product Management xzhong@qti.qualcomm.com For access to the SDK contact: © 2022 Qualcomm
- 19. Thank You 19