Skip to content

image-labeling

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

image-labeling

README.md

Capacitor ML Kit Image Labeling Plugin

Unofficial Capacitor plugin for ML Kit Image Labeling.1

Use Cases

The Image Labeling plugin is typically used whenever an app needs to understand the content of an image, for example:

  • Content categorization: Automatically tag and organize photos by the objects, places, or activities they contain.
  • Accessibility: Generate descriptive labels for images to support screen readers and other assistive technologies.
  • Content moderation: Detect the presence of certain objects or scenes before an image is published.
  • Search and discovery: Make an image library searchable by the labels detected in each image.

Compatibility

Plugin Version Capacitor Version Status
8.x.x >=8.x.x Active support

Installation

You can use our AI-Assisted Setup to install the plugin. Add the Capawesome Skills to your AI tool using the following command:

npx skills add capawesome-team/skills --skill capacitor-plugins

Then use the following prompt:

Use the `capacitor-plugins` skill from `capawesome-team/skills` to install the `@capacitor-mlkit/image-labeling` plugin in my project.

If you prefer Manual Setup, install the plugin by running the following commands and follow the platform-specific instructions below:

npm install @capacitor-mlkit/image-labeling
npx cap sync

Attention: This plugin only supports CocoaPods for iOS dependency management. Swift Package Manager (SPM) is not supported for the ML Kit SDK, see this comment.

Android

Variables

If needed, you can define the following project variable in your app’s variables.gradle file to change the default version of the dependency:

  • $mlkitImageLabelingVersion version of com.google.mlkit:image-labeling (default: 17.0.9)

This can be useful if you encounter dependency conflicts with other plugins in your project.

iOS

Minimum Deployment Target

Make sure to set the deployment target in your ios/App/Podfile to at least 15.5:

platform :ios, '15.5'

Configuration

No configuration required for this plugin.

Demo

A working example can be found here: robingenz/capacitor-mlkit-plugin-demo

Usage

The following example shows how to detect labels in an image.

Detect labels in an image

Detect labels in an image at a local path. You can configure the confidence threshold to control which labels are returned. Only available on Android and iOS:

import { ImageLabeling } from '@capacitor-mlkit/image-labeling';

const processImage = async () => {
  const { labels } = await ImageLabeling.processImage({
    path: 'path/to/image.jpg',
    confidenceThreshold: 0.5,
  });
  return labels;
};

API

processImage(...)

processImage(options: ProcessImageOptions) => Promise<ProcessImageResult>

Detects labels in the supplied image.

Labels can describe objects, locations, activities, animal species, products and more.

Only available on Android and iOS.

Param Type
options ProcessImageOptions

Returns: Promise<ProcessImageResult>

Since: 8.2.0

Interfaces

ProcessImageResult

Prop Type Description Since
labels ImageLabel[] The detected labels. 8.2.0

ImageLabel

Represents a label detected in an image.

Prop Type Description Since
index number The index of the label within the model. 8.2.0
text string The text of the label (e.g. Bicycle). The text is always in English. 8.2.0
confidence number The confidence of the label. The value is between 0.0 and 1.0. 8.2.0

ProcessImageOptions

Prop Type Description Default Since
path string The local path to the image file. 8.2.0
confidenceThreshold number The confidence threshold for the labels. Only labels with a confidence greater than or equal to this value are returned. The value must be between 0.0 and 1.0. 0.5 8.2.0

FAQ

Which platforms are supported by this plugin?

The processImage(...) method is only available on Android and iOS. The Web platform is not supported by the underlying ML Kit Image Labeling SDK.

Which model does the plugin use?

The plugin uses the base image labeling model, which is bundled with the app and can recognize more than 400 labels covering objects, places, activities, animal species, products, and more. Custom TensorFlow Lite models are currently not supported.

What does the confidence threshold do?

The confidenceThreshold option controls which labels are returned. Only labels with a confidence greater than or equal to the threshold are included in the result. The value must be between 0.0 and 1.0 and defaults to 0.5.

In which language are the labels returned?

The label text is always returned in English. Use the index property of a label together with your own localized lookup table if you need translated labels.

Can I use this plugin with Ionic, React, Vue or Angular?

Yes, the plugin is framework-agnostic. It works in any Capacitor app regardless of the web framework, including Ionic with Angular, React, or Vue, as well as plain JavaScript projects.

Related Plugins

Terms & Privacy

This plugin uses the Google ML Kit:

Newsletter

Stay up to date with the latest news and updates about the Capawesome, Capacitor, and Ionic ecosystem by subscribing to our Capawesome Newsletter.

Changelog

See CHANGELOG.md.

License

See LICENSE.

Footnotes

  1. This project is not affiliated with, endorsed by, sponsored by, or approved by Google LLC or any of their affiliates or subsidiaries. ↩