What’s new in VisionKit

This API is all about lifting subjects from images, i.e. pulling out your favorite pet from a picture to be used elsewhere 

Subject Lifting

  • Simple long press on an image, you can share the subject.  Now you can lift any subject and turn it into a sticker.  This is automatically for most code from last year.
    • Just set an appropriate interaction type, .automatic does a combination of al types
    • .imageSegmentation will only do Subject lifting
    • .automaticTextOnly – same as iOS 16 behavior – test selection and data detectors

Visual Look Up

  • Allows users to know about pets, nature, landmarks, art and Media.  In iOS17 adding Food, Products and Signs & Symbols
    • This helps with laundry tag information!
    • Works for 6 languages to start with.
  • Analysis is entirely on the device. Includes feature extraction.
    • One you look up the object  then it goes to server for more information.
    • Add .visualLookup to your analyzer configuration 
    • Badges are displayed on subjects for lookup etc. 

Data Scanner and Live Text – allows for OCR 

  • Optical flow – can now do high speed scanning and comes for free when recognizing text.  
  • Currency Support
    • Find and interact with monetary by setting textContentType to .currency in the initializer, just like email or phone numbers
    • Provides both a Bounds and a transcript.  Code example in video for totaling a receipt.
  • Live Text – more regions:
    • Can detect document structure, like list detection (iOS16), but now can also detect Tables   Will even merge cells if needed.
    • Context-aware data detectors – will pull additional information from surrounding areas, like name and address
    • You can do .transcript last year, you can now do ranges, selected text, Attributed and identify when text changes.

Expanded platform support

  • All about the Mac – Catalyst support being rolled out
  • Catalyst –
    • Simple recompile for LiveText, Subject Lifting and Visual Look Up, no QR support yet ( you can do it via Vision Framework)
  • Native Mac API
    • ImageAnalyzer
    • ImageAnalysisOverlayView – subclass of NSView, just add above image content (adding as SubView of ContentView is simpler)
    • Basically the name as iOS (except .machine-readable is a NOOP
    • Contents rect – needs to know the rectangle bounds that it needs.
    • If you use NSView you can just set trackingImageView on the overlay and the system will do it for you
  • Contextual Menus (Mac)
    • You can combine items in the same menu, this provides a much better right click menus based on selected item.
    • Items are identified by tags, there is a struct that defines them.  Code examples are in the session.