Skip to content

Latest commit

 

History

History
391 lines (310 loc) · 9.58 KB

README.md

File metadata and controls

391 lines (310 loc) · 9.58 KB
Raw

@nativescript/mlkit-core

A plugin that provides a UI component to access the different functionalities of Google's ML Kit SDK.

Contents

Installation

npm install @nativescript/mlkit-core

Use @nativescript/mlkit-core

The usage of @nativescript/mlkit-core has the following flow:

  1. Registering and adding MLKitView to your markup.

  2. Setting the detectionType and listening to the detection event.

To access all the vision APIs at once, set the detectionType property to 'all' and identify them in the detection event's handler.

To access a specific API, Barcode scanning for example, set the detectionType property to the API name ('barcode' for Barcode scanning), AND import that API's NativeScript plugin(@nativescript/mlkit-barcode-scanning).

  1. Check if ML Kit is supported To verify if ML Kit is supported on the device, call the static isAvailable() method on MLKitView class.
if(MLKitView.isAvailable()){

}
  1. Request for permission to access the device camera by calling requestCameraPermission():
mlKitView.requestCameraPermission().then(()=>{

})

The following are examples of registering and using MLKitView in the different JS flavors.

Core

  1. Register MLKitView by adding xmlns:ui="@nativescript/mlkit-core" to the Page element.

  2. Use the ui prefix to access MLKitView from the plugin.

<ui:MLKitView
  cameraPosition="back"
   detectionType="all" 
   detection="onDetection"
/>

Angular

  1. In Angular, register the MLKitView by adding MLKitModule to the NgModule of the component where you want to use MLKitView.
import { MLKitModule } from '@nativescript/mlkit-core/angular';

@NgModule({
    imports: [
    MLKitModule
    ],
    declarations: [
        AppComponent
    ],
    bootstrap: [AppComponent]
})
  1. Use MLKitView in markup.
<MLKitView
cameraPosition="back"
detectionType="all" 
(detection)="onDetection($event)"
></MLKitView>

Vue

  1. To use MLKitView, register it in the app.ts by passing it to the use method of the app instance.
import { createApp } from 'nativescript-vue'

import MLKit from '@nativescript/mlkit-core/vue'
import Home from './components/Home.vue';

const app = createApp(Home)

app.use(MLKit)
  1. Use MLKitView in markup.
<MLKitView
cameraPosition="back"
detectionType="all" 
@detection="onDetection"
/>

Vision APIs optional modules

Important: Detection works only for optional modules installed

Barcode Scanning

npm i @nativescript/mlkit-barcode-scanning

import { DetectionType, DetectionEvent } from '@nativescript/mlkit-core';
import { BarcodeResult } from '@nativescript/mlkit-barcode-scanning';
onDetection(event: DetectionEvent){
    if(event.type === DetectionType.Barcode){
        const barcode: BarcodeResult[] = event.data;
    }
}

For more details, see @nativescript/mlkit-barcode-scanning

Face Detection

npm install @nativescript/mlkit-face-detection

import { DetectionType, DetectionEvent } from '@nativescript/mlkit-core';
import { FaceResult } from '@nativescript/mlkit-face-detection';

onDetection(event: DetectionEvent){
    if(event.type === DetectionType.Face){
        const faces: FaceResult[] = event.data;
    }
}

For more details, see @nativescript/mlkit-face-detection

Image Labeling

npm install @nativescript/mlkit-image-labeling

import { DetectionType, DetectionEvent } from '@nativescript/mlkit-core';
import { ImageLabelingResult } from '@nativescript/mlkit-image-labeling';
onDetection(event: DetectionEvent){
    if(event.type === DetectionType.Image){
        const labels: ImageLabelingResult[] = event.data;
    }
}

For more details, see nativescript/mlkit-image-labeling

Object Detection

npm install @nativescript/mlkit-object-detection

import { DetectionType, DetectionEvent } from '@nativescript/mlkit-core';
import { ObjectResult } from '@nativescript/mlkit-object-detection'
onDetection(event: DetectionEvent){
    if(event.type === DetectionType.Object){
        const objects: ObjectResult[] = event.data;
    }
}

For more details, see @nativescript/mlkit-object-detection

Pose Detection

npm install @nativescript/mlkit-pose-detection

import { DetectionType, DetectionEvent } from '@nativescript/mlkit-core';
import { PoseResult } from '@nativescript/mlkit-pose-detection';
onDetection(event: DetectionEvent){
    if(event.type === DetectionType.Pose){
        const poses: PoseResult = event.data;
    }
}

For more details, see @nativescript/mlkit-pose-detection

Text Recognition

npm install @nativescript/mlkit-text-recognition

import { DetectionType, DetectionEvent } from '@nativescript/mlkit-core';
import { TextResult } from '@nativescript/mlkit-text-recognition';
onDetection(event: DetectionEvent){
    if(event.type === DetectionType.Text){
        const text: TextResult = event.data;
    }
}

For more details, see @nativescript/mlkit-text-recognition

API

detectWithStillImage()

import { DetectionType, detectWithStillImage } from "@nativescript/mlkit-core";

async processStill(args) {
        try {
           
            result: { [key: string]: any } = await detectWithStillImage(image: ImageSource, options)
        } catch (e) {
            console.log(e);
        }
    }

Detects barcode, pose, etc from a still image instead of using the camera.

  • image: The image to detect the object from
  • options: An optional StillImageDetectionOptions object parameter specifying the detection characteristics.

MLKitView class

The MLKitView class provides the camera view for detection.

It has the following members.

Properties

Property Type
detectionEvent string
cameraPosition CameraPosition
detectionType DetectionType
barcodeFormats BarcodeFormats
faceDetectionPerformanceMode FaceDetectionPerformanceMode
faceDetectionTrackingEnabled boolean
faceDetectionMinFaceSize number
imageLabelerConfidenceThreshold number
objectDetectionMultiple boolean
objectDetectionClassify boolean
torchOn boolean
pause boolean
processEveryNthFrame number
readonly latestImage? ImageSource
retrieveLatestImage boolean

Methods

Method Returns Description
isAvailable() boolean A static method to check if the device supports ML Kit.
stopPreview() void
startPreview() void
toggleCamera() void
requestCameraPermission() Promise<void>
hasCameraPermission() boolean
on() void

StillImageDetectionOptions interface

interface StillImageDetectionOptions {
  detectorType: DetectionType;

  barcodeScanning?: {
    barcodeFormat?: [BarcodeFormats];
  };
  faceDetection?: {
    faceTracking?: boolean;
    minimumFaceSize: ?number;
    detectionMode?: 'fast' | 'accurate';
    landmarkMode?: 'all' | 'none';
    contourMode?: 'all' | 'none';
    classificationMode?: 'all' | 'none';
  };
  imageLabeling?: {
    confidenceThreshold?: number;
  };
  objectDetection?: {
    multiple: boolean;
    classification: boolean;
  };
  selfieSegmentation?: {
    enableRawSizeMask?: boolean;
    smoothingRatio?: number;
  };
}

Enums

DetectionType

export enum DetectionType {
  Barcode = 'barcode',
  DigitalInk = 'digitalInk',
  Face = 'face',
  Image = 'image',
  Object = 'object',
  Pose = 'pose',
  Text = 'text',
  All = 'all',
  Selfie = 'selfie',
  None = 'none',
}

CameraPosition

export enum CameraPosition {
  FRONT = 'front',
  BACK = 'back',
}

BarcodeFormats

export enum BarcodeFormats {
  ALL = 'all',
  CODE_128 = 'code_128',
  CODE_39 = 'code_39',
  CODE_93 = 'code_93',
  CODABAR = 'codabar',
  DATA_MATRIX = 'data_matrix',
  EAN_13 = 'ean_13',
  EAN_8 = 'ean_8',
  ITF = 'itf',
  QR_CODE = 'qr_code',
  UPC_A = 'upc_a',
  UPC_E = 'upc_e',
  PDF417 = 'pdf417',
  AZTEC = 'aztec',
  UNKOWN = 'unknown',
}

FaceDetectionPerformanceMode

export enum FaceDetectionPerformanceMode {
  Fast = 'fast',
  Accurate = 'accurate',
}

License

Apache License Version 2.0