Skip to content

Commit 3fe1cae

Browse files
jakmroNorbertKlockiewiczchmjkb
authored
docs: Add image classification docs (#65)
## Description Add useClassification docs. ### Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] Documentation update (improves or adds clarity to existing documentation) ### Checklist - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [ ] My changes generate no new warnings --------- Co-authored-by: Norbert Klockiewicz <[email protected]> Co-authored-by: chmjkb <[email protected]>
1 parent aec95c6 commit 3fe1cae

File tree

6 files changed

+156
-163
lines changed

6 files changed

+156
-163
lines changed

‎docs/docs/computer-vision/useClassification.mdx

+102
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
---
2+
title: useClassification
3+
sidebar_position: 1
4+
---
5+
6+
Image classification is the process of assigning a label to an image that best describes its contents. For example, when given an image of a puppy, the image classifier should assign the puppy class to that image.
7+
8+
:::info
9+
Usually, the class with the highest probability is the one that is assigned to an image. However, if there are multiple classes with comparatively high probabilities, this may indicate that the model is not confident in its prediction.
10+
:::
11+
12+
:::caution
13+
It is recommended to use models provided by us, which are available at our [Hugging Face repository](https://huggingface.co/software-mansion/react-native-executorch-efficientnet-v2-s). You can also use [constants](https://github.com/software-mansion/react-native-executorch/tree/main/src/constants/modelUrls.ts) shipped with our library
14+
:::
15+
16+
## Reference
17+
18+
```typescript
19+
import { useClassification, EFFICIENTNET_V2_S } from 'react-native-executorch';
20+
21+
const model = useClassification({
22+
modelSource: EFFICIENTNET_V2_S,
23+
});
24+
25+
const imageUri = 'file::///Users/.../cute_puppy.png';
26+
27+
try {
28+
const classesWithProbabilities = await model.forward(imageUri);
29+
} catch (error) {
30+
console.error(error);
31+
}
32+
```
33+
34+
<details>
35+
<summary>Type definitions</summary>
36+
37+
```typescript
38+
interface ClassificationModule {
39+
error: string | null;
40+
isReady: boolean;
41+
isGenerating: boolean;
42+
forward: (input: string) => Promise<{ [category: string]: number }>;
43+
}
44+
```
45+
46+
</details>
47+
48+
### Arguments
49+
50+
**`modelSource`**
51+
A string that specifies the location of the model binary. For more information, take a look at [loading models](../fundamentals/loading-models.md) page.
52+
53+
### Returns
54+
55+
| Field | Type | Description |
56+
| -------------- | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
57+
| `forward` | `(input: string) => Promise<{ [category: string]: number }>` | Executes the model's forward pass, where `input` can be a fetchable resource or a Base64-encoded string. |
58+
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load. |
59+
| `isGenerating` | `boolean` | Indicates whether the model is currently processing an inference. |
60+
| `isReady` | `boolean` | Indicates whether the model has successfully loaded and is ready for inference. |
61+
62+
## Running the model
63+
64+
To run the model, you can use the `forward` method. It accepts one argument, which is the image. The image can be a remote URL, a local file URI, or a base64-encoded image. The function returns a promise, which can resolve either to error or an object containing categories with their probabilities.
65+
66+
:::info[Info]
67+
Images from external sources are stored in your application's temporary directory.
68+
:::
69+
70+
## Example
71+
72+
```typescript
73+
import { useClassification, EFFICIENTNET_V2_S } from 'react-native-executorch';
74+
75+
function App() {
76+
const model = useClassification({
77+
modulePath: EFFICIENTNET_V2_S,
78+
});
79+
80+
...
81+
const imageUri = 'file:///Users/.../cute_puppy.png';
82+
83+
try {
84+
const classesWithProbabilities = await model.forward(imageUri);
85+
86+
// Extract three classes with the highest probabilities
87+
const topThreeClasses = Object.entries(classesWithProbabilities)
88+
.sort(([, a], [, b]) => b - a)
89+
.slice(0, 3)
90+
.map(([label, score]) => ({ label, score }));
91+
} catch (error) {
92+
console.error(error);
93+
}
94+
...
95+
}
96+
```
97+
98+
## Supported models
99+
100+
| Model | Number of classes | Class list |
101+
| --------------------------------------------------------------------------------------------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
102+
| [efficientnet_v2_s](https://pytorch.org/vision/0.20/models/generated/torchvision.models.efficientnet_v2_s.html) | 1000 | [ImageNet1k_v1](https://github.com/software-mansion/react-native-executorch/blob/main/android/src/main/java/com/swmansion/rnexecutorch/models/classification/Constants.kt) |

‎docs/docs/computer-vision/useObjectDetection.mdx

+53-26
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,22 @@
11
---
22
title: useObjectDetection
3-
sidebar_position: 1
3+
sidebar_position: 2
44
---
55

6-
`useObjectDetection` is a hook that lets you seamlessly integrate object detection into your React Native application. Currently, the SSDLite320Large model with MobileNetv3 backbone is supported.
6+
Object detection is a computer vision technique that identifies and locates objects within images or video. It’s commonly used in applications like image recognition, video surveillance or autonomous driving.
7+
`useObjectDetection` is a hook that allows you to seamlessly integrate object detection into your React Native applications.
8+
9+
:::caution
10+
It is recommended to use models provided by us, which are available at our [Hugging Face repository](https://huggingface.co/software-mansion/react-native-executorch-ssdlite320-mobilenet-v3-large). You can also use [constants](https://github.com/software-mansion/react-native-executorch/blob/69802ee1ca161d9df00def1dabe014d36341cfa9/src/constants/modelUrls.ts#L28) shipped with our library.
11+
:::
712

813
## Reference
914
```jsx
10-
import { useObjectDetection } from 'react-native-executorch';
15+
import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE } from 'react-native-executorch';
1116

1217
function App() {
1318
const ssdlite = useObjectDetection({
14-
modelSource: require("./assets/ssdlite320large_mobilenetv3.pte"),
19+
modelSource: SSDLITE_320_MOBILENET_V3_LARGE, // alternatively, you can use require(...)
1520
});
1621

1722
...
@@ -28,30 +33,53 @@ function App() {
2833
<summary>Type definitions</summary>
2934

3035
```typescript
36+
interface Bbox {
37+
x1: number;
38+
x2: number;
39+
y1: number;
40+
y2: number;
41+
}
42+
43+
interface Detection {
44+
bbox: Bbox;
45+
label: keyof typeof CocoLabel;
46+
score: number;
47+
}
48+
49+
interface ObjectDetectionModule {
50+
error: string | null;
51+
isReady: boolean;
52+
isGenerating: boolean;
53+
forward: (input: string) => Promise<Detection[]>;
54+
}
3155
```
3256
</details>
3357

3458
### Arguments
3559

3660
`modelSource`
3761

38-
A String that specifies the path to the model file. You can download the model from our HuggingFace repository.
39-
For SSDLite, you can add it to your assets directory, and use `require()`. If you prefer to download the model
40-
the model in runtime instead of bundling it, you can use the constants that we ship with the library.
62+
A string that specifies the path to the model file. You can download the model from our [HuggingFace repository](https://huggingface.co/software-mansion/react-native-executorch-ssdlite320-mobilenet-v3-large/tree/main).
63+
For more information on that topic, you can check out the [Loading models](https://docs.swmansion.com/react-native-executorch/fundamentals/loading-models) page.
4164

4265
### Returns
4366

4467
The hook returns an object with the following properties:
4568

4669

47-
| Field | Type | Description |
48-
| ------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------- |
49-
| `forward` | `(input: string) => Promise<Detection[]>` | Function that accepts an image and returns an array of Detection objects |
50-
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load or failed during generation process |
51-
| `isGenerating` | `boolean` | Indicates whether the model is processing the response |
52-
| `isReady` | `boolean` | Indicates whether the model has properly loaded and is ready for inference |
70+
| **Field** | **Type** | **Description** |
71+
|-----------------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------|
72+
| `forward` | `(input: string) => Promise<Detection[]>` | A function that accepts an image (url, b64) and returns an array of `Detection` objects. |
73+
| `error` | <code>string &#124; null</code> | Contains the error message if the model loading failed. |
74+
| `isGenerating` | `boolean` | Indicates whether the model is currently processing an inference. |
75+
| `isReady` | `boolean` | Indicates whether the model has successfully loaded and is ready for inference. |
5376

54-
### Detection object
77+
78+
## Running the model
79+
80+
To run the model, you can use the `forward` method. It accepts one argument, which is the image. The image can be a remote URL, a local file URI, or a base64-encoded image. The function returns an array of `Detection` objects. Each object contains coordinates of the bounding box, the label of the detected object, and the confidence score. For more information, please refer to the reference or type definitions.
81+
82+
## Detection object
5583
The detection object is specified as follows:
5684
```typescript
5785
interface Bbox {
@@ -67,34 +95,33 @@ interface Detection {
6795
score: number;
6896
}
6997
```
70-
The `bbox` property contains information about the bounding box of detected objects. It is represented as two points, one on the left bottom part of the bounding box (x1, y1), the second one as the top right part (x2, y2).
71-
The label property contains the name of the detected object, which is one of `CocoLabels`. The `score` is a confidence score of the detected object.
98+
The `bbox` property contains information about the bounding box of detected objects. It is represented as two points: one at the bottom-left corner of the bounding box (`x1`, `y1`) and the other at the top-right corner (`x2`, `y2`).
99+
The `label` property contains the name of the detected object, which corresponds to one of the `CocoLabels`. The `score` represents the confidence score of the detected object.
72100

73-
### Running the model
74101

75-
To run the model, you can use the `forward` method. It accepts one argument, which is the image. It can be either a remote URL,
76-
a local file or base64 encoded image. The function returns an array of `Detection` objects. Each one contains coordinates
77-
of the bounding box, the label of the detected object and confidence score. For more information, please refer to the reference or example.
78102

79-
### End to end example
103+
## Example
80104
```tsx
81-
import { useObjectDetection, SSDLITE320LARGE_MOBILENETV3_WEIGHTS } from 'react-native-executorch';
105+
import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE } from 'react-native-executorch';
82106

83107
function App() {
84108
const ssdlite = useObjectDetection({
85-
modelSource: SSDLITE320LARGE_MOBILENETV3_WEIGHTS, // Can also use require('') as well
109+
modelSource: SSDLITE_320_MOBILENET_V3_LARGE,
86110
});
87111

88112
const runModel = async () => {
89113
const detections = await ssdlite.forward("https://url-to-image.jpg");
90114
for (const detection of detections) {
91-
console.log("Bounding box: ", detection.bbox); // [x, y, width, height]
115+
console.log("Bounding box: ", detection.bbox);
92116
console.log("Bounding label: ", detection.label);
93117
console.log("Bounding score: ", detection.score);
94118
}
95119
}
96120
}
97121
```
98122

99-
### Benchmarks
100-
TODO
123+
## Supported Models
124+
125+
| Model | Number of classes | Class list |
126+
| --------------------------------------------------------------------------------------------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
127+
| [SSDLite320 MobileNetV3 Large](https://pytorch.org/vision/main/models/generated/torchvision.models.detection.ssdlite320_mobilenet_v3_large.html#torchvision.models.detection.SSDLite320_MobileNet_V3_Large_Weights) | 91 | [COCO](https://github.com/software-mansion/react-native-executorch/blob/69802ee1ca161d9df00def1dabe014d36341cfa9/src/types/object_detection.ts#L14) |

‎docs/docs/computer-vision/useStyleTransfer.md ‎docs/docs/computer-vision/useStyleTransfer.mdx

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: useStyleTransfer
3-
sidebar_position: 1
3+
sidebar_position: 3
44
---
55

66
Style transfer is a technique used in computer graphics and machine learning where the visual style of one image is applied to the content of another. This is achieved using algorithms that manipulate data from both images, typically with the aid of a neural network. The result is a new image that combines the artistic elements of one picture with the structural details of another, effectively merging art with traditional imagery. React Native ExecuTorch offers a dedicated hook `useStyleTransfer`, for this task. However before you start you'll need to obtain ExecuTorch-compatible model binary.

‎docs/docs/computer_vision/_category_.json

-8
This file was deleted.

0 commit comments

Comments
 (0)