This API is all about lifting subjects from images, i.e. pulling out your favorite pet from a picture to be used elsewhere
Subject Lifting
- Simple long press on an image, you can share the subject. Now you can lift any subject and turn it into a sticker. This is automatically for most code from last year.
- Just set an appropriate interaction type, .automatic does a combination of al types
- .imageSegmentation will only do Subject lifting
- .automaticTextOnly – same as iOS 16 behavior – test selection and data detectors
Visual Look Up
- Allows users to know about pets, nature, landmarks, art and Media. In iOS17 adding Food, Products and Signs & Symbols
- This helps with laundry tag information!
- Works for 6 languages to start with.
- Analysis is entirely on the device. Includes feature extraction.
- One you look up the object then it goes to server for more information.
- Add .visualLookup to your analyzer configuration
- Badges are displayed on subjects for lookup etc.
Data Scanner and Live Text – allows for OCR
- Optical flow – can now do high speed scanning and comes for free when recognizing text.
- Currency Support
- Find and interact with monetary by setting textContentType to .currency in the initializer, just like email or phone numbers
- Provides both a Bounds and a transcript. Code example in video for totaling a receipt.
- Live Text – more regions:
- Can detect document structure, like list detection (iOS16), but now can also detect Tables Will even merge cells if needed.
- Context-aware data detectors – will pull additional information from surrounding areas, like name and address
- You can do .transcript last year, you can now do ranges, selected text, Attributed and identify when text changes.
Expanded platform support
- All about the Mac – Catalyst support being rolled out
- Catalyst –
- Simple recompile for LiveText, Subject Lifting and Visual Look Up, no QR support yet ( you can do it via Vision Framework)
- Native Mac API
- ImageAnalyzer
- ImageAnalysisOverlayView – subclass of NSView, just add above image content (adding as SubView of ContentView is simpler)
- Basically the name as iOS (except .machine-readable is a NOOP
- Contents rect – needs to know the rectangle bounds that it needs.
- If you use NSView you can just set trackingImageView on the overlay and the system will do it for you
- Contextual Menus (Mac)
- You can combine items in the same menu, this provides a much better right click menus based on selected item.
- Items are identified by tags, there is a struct that defines them. Code examples are in the session.