Skip to content

Commit cbe2b55

Browse files
authored
feat: Hookless API (#82)
## Description This PR introduces hookless API and restructures the `./src` directory. ### Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update (improves or adds clarity to existing documentation) ### Tested on - [x] iOS - [x] Android ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have updated the documentation accordingly - [x] My changes generate no new warnings
1 parent 7d391c8 commit cbe2b55

23 files changed

+320
-133
lines changed

docs/docs/computer-vision/useClassification.mdx

-14
Original file line numberDiff line numberDiff line change
@@ -31,20 +31,6 @@ try {
3131
}
3232
```
3333

34-
<details>
35-
<summary>Type definitions</summary>
36-
37-
```typescript
38-
interface ClassificationModule {
39-
error: string | null;
40-
isReady: boolean;
41-
isGenerating: boolean;
42-
forward: (input: string) => Promise<{ [category: string]: number }>;
43-
}
44-
```
45-
46-
</details>
47-
4834
### Arguments
4935

5036
**`modelSource`**

docs/docs/computer-vision/useObjectDetection.mdx

+26-27
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ It is recommended to use models provided by us, which are available at our [Hugg
1111
:::
1212

1313
## Reference
14+
1415
```jsx
1516
import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE } from 'react-native-executorch';
1617

@@ -45,42 +46,36 @@ interface Detection {
4546
label: keyof typeof CocoLabel;
4647
score: number;
4748
}
48-
49-
interface ObjectDetectionModule {
50-
error: string | null;
51-
isReady: boolean;
52-
isGenerating: boolean;
53-
forward: (input: string) => Promise<Detection[]>;
54-
}
5549
```
50+
5651
</details>
5752

5853
### Arguments
5954

6055
`modelSource`
6156

62-
A string that specifies the path to the model file. You can download the model from our [HuggingFace repository](https://huggingface.co/software-mansion/react-native-executorch-ssdlite320-mobilenet-v3-large/tree/main).
57+
A string that specifies the path to the model file. You can download the model from our [HuggingFace repository](https://huggingface.co/software-mansion/react-native-executorch-ssdlite320-mobilenet-v3-large/tree/main).
6358
For more information on that topic, you can check out the [Loading models](https://docs.swmansion.com/react-native-executorch/fundamentals/loading-models) page.
6459

6560
### Returns
6661

6762
The hook returns an object with the following properties:
6863

69-
70-
| **Field** | **Type** | **Description** |
71-
|-----------------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------|
72-
| `forward` | `(input: string) => Promise<Detection[]>` | A function that accepts an image (url, b64) and returns an array of `Detection` objects. |
73-
| `error` | <code>string &#124; null</code> | Contains the error message if the model loading failed. |
74-
| `isGenerating` | `boolean` | Indicates whether the model is currently processing an inference. |
75-
| `isReady` | `boolean` | Indicates whether the model has successfully loaded and is ready for inference. |
76-
64+
| Field | Type | Description |
65+
| -------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------- |
66+
| `forward` | `(input: string) => Promise<Detection[]>` | A function that accepts an image (url, b64) and returns an array of `Detection` objects. |
67+
| `error` | <code>string &#124; null</code> | Contains the error message if the model loading failed. |
68+
| `isGenerating` | `boolean` | Indicates whether the model is currently processing an inference. |
69+
| `isReady` | `boolean` | Indicates whether the model has successfully loaded and is ready for inference. |
7770

7871
## Running the model
7972

8073
To run the model, you can use the `forward` method. It accepts one argument, which is the image. The image can be a remote URL, a local file URI, or a base64-encoded image. The function returns an array of `Detection` objects. Each object contains coordinates of the bounding box, the label of the detected object, and the confidence score. For more information, please refer to the reference or type definitions.
8174

8275
## Detection object
76+
8377
The detection object is specified as follows:
78+
8479
```typescript
8580
interface Bbox {
8681
x1: number;
@@ -95,33 +90,37 @@ interface Detection {
9590
score: number;
9691
}
9792
```
93+
9894
The `bbox` property contains information about the bounding box of detected objects. It is represented as two points: one at the bottom-left corner of the bounding box (`x1`, `y1`) and the other at the top-right corner (`x2`, `y2`).
9995
The `label` property contains the name of the detected object, which corresponds to one of the `CocoLabels`. The `score` represents the confidence score of the detected object.
10096

101-
102-
10397
## Example
98+
10499
```tsx
105-
import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE } from 'react-native-executorch';
100+
import {
101+
useObjectDetection,
102+
SSDLITE_320_MOBILENET_V3_LARGE,
103+
} from 'react-native-executorch';
106104

107105
function App() {
108106
const ssdlite = useObjectDetection({
109107
modelSource: SSDLITE_320_MOBILENET_V3_LARGE,
110108
});
111109

112110
const runModel = async () => {
113-
const detections = await ssdlite.forward("https://url-to-image.jpg");
111+
const detections = await ssdlite.forward('https://url-to-image.jpg');
112+
114113
for (const detection of detections) {
115-
console.log("Bounding box: ", detection.bbox);
116-
console.log("Bounding label: ", detection.label);
117-
console.log("Bounding score: ", detection.score);
114+
console.log('Bounding box: ', detection.bbox);
115+
console.log('Bounding label: ', detection.label);
116+
console.log('Bounding score: ', detection.score);
118117
}
119-
}
118+
};
120119
}
121120
```
122121

123122
## Supported models
124123

125-
| Model | Number of classes | Class list |
126-
| --------------------------------------------------------------------------------------------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
127-
| [SSDLite320 MobileNetV3 Large](https://pytorch.org/vision/main/models/generated/torchvision.models.detection.ssdlite320_mobilenet_v3_large.html#torchvision.models.detection.SSDLite320_MobileNet_V3_Large_Weights) | 91 | [COCO](https://github.com/software-mansion/react-native-executorch/blob/69802ee1ca161d9df00def1dabe014d36341cfa9/src/types/object_detection.ts#L14) |
124+
| Model | Number of classes | Class list |
125+
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
126+
| [SSDLite320 MobileNetV3 Large](https://pytorch.org/vision/main/models/generated/torchvision.models.detection.ssdlite320_mobilenet_v3_large.html#torchvision.models.detection.SSDLite320_MobileNet_V3_Large_Weights) | 91 | [COCO](https://github.com/software-mansion/react-native-executorch/blob/69802ee1ca161d9df00def1dabe014d36341cfa9/src/types/object_detection.ts#L14) |

docs/docs/computer-vision/useStyleTransfer.mdx

-14
Original file line numberDiff line numberDiff line change
@@ -30,20 +30,6 @@ try {
3030
}
3131
```
3232

33-
<details>
34-
<summary>Type definitions</summary>
35-
36-
```typescript
37-
interface StyleTransferModule {
38-
error: string | null;
39-
isReady: boolean;
40-
isGenerating: boolean;
41-
forward: (input: string) => Promise<string>;
42-
}
43-
```
44-
45-
</details>
46-
4733
### Arguments
4834

4935
**`modelSource`**

examples/computer-vision/components/ImageWithBboxes.tsx

+2-7
Original file line numberDiff line numberDiff line change
@@ -68,13 +68,7 @@ export default function ImageWithBboxes({
6868
const height = (y2 - y1) * scaleY;
6969

7070
return (
71-
<View
72-
key={index}
73-
style={[
74-
styles.bbox,
75-
{ left, top, width, height, borderColor: 'red' },
76-
]}
77-
>
71+
<View key={index} style={[styles.bbox, { left, top, width, height }]}>
7872
<Text style={styles.label}>
7973
{detection.label} ({(detection.score * 100).toFixed(1)}%)
8074
</Text>
@@ -98,6 +92,7 @@ const styles = StyleSheet.create({
9892
bbox: {
9993
position: 'absolute',
10094
borderWidth: 2,
95+
borderColor: 'red',
10196
},
10297
label: {
10398
position: 'absolute',

examples/computer-vision/screens/ObjectDetectionScreen.tsx

+5-1
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ export const ObjectDetectionScreen = ({
7676
/>
7777
) : (
7878
<Image
79-
style={{ width: '100%', height: '100%' }}
79+
style={styles.fullSizeImage}
8080
resizeMode="contain"
8181
source={require('../assets/icons/executorch_logo.png')}
8282
/>
@@ -127,4 +127,8 @@ const styles = StyleSheet.create({
127127
flex: 1,
128128
marginRight: 4,
129129
},
130+
fullSizeImage: {
131+
width: '100%',
132+
height: '100%',
133+
},
130134
});

examples/llama/screens/ChatScreen.tsx

+4-1
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ export default function ChatScreen() {
6363
<SafeAreaView style={styles.container}>
6464
<TouchableWithoutFeedback onPress={Keyboard.dismiss}>
6565
<KeyboardAvoidingView
66-
style={{ flex: 1 }}
66+
style={styles.keyboardAvoidingView}
6767
behavior={Platform.OS === 'ios' ? 'padding' : 'height'}
6868
keyboardVerticalOffset={Platform.OS === 'android' ? 30 : 0}
6969
>
@@ -133,6 +133,9 @@ const styles = StyleSheet.create({
133133
container: {
134134
flex: 1,
135135
},
136+
keyboardAvoidingView: {
137+
flex: 1,
138+
},
136139
topContainer: {
137140
height: 68,
138141
width: '100%',

src/models/Classification.ts src/hooks/computer_vision/useClassification.ts

+6-8
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,19 @@
11
import { useState } from 'react';
2-
import { _ClassificationModule } from '../native/RnExecutorchModules';
3-
import { useModule } from '../useModule';
2+
import { _ClassificationModule } from '../../native/RnExecutorchModules';
3+
import { useModule } from '../../useModule';
44

55
interface Props {
66
modelSource: string | number;
77
}
88

9-
interface ClassificationModule {
9+
export const useClassification = ({
10+
modelSource,
11+
}: Props): {
1012
error: string | null;
1113
isReady: boolean;
1214
isGenerating: boolean;
1315
forward: (input: string) => Promise<{ [category: string]: number }>;
14-
}
15-
16-
export const useClassification = ({
17-
modelSource,
18-
}: Props): ClassificationModule => {
16+
} => {
1917
const [module, _] = useState(() => new _ClassificationModule());
2018
const {
2119
error,

src/models/ObjectDetection.ts src/hooks/computer_vision/useObjectDetection.ts

+7-9
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,20 @@
11
import { useState } from 'react';
2-
import { _ObjectDetectionModule } from '../native/RnExecutorchModules';
3-
import { useModule } from '../useModule';
4-
import { Detection } from '../types/object_detection';
2+
import { _ObjectDetectionModule } from '../../native/RnExecutorchModules';
3+
import { useModule } from '../../useModule';
4+
import { Detection } from '../../types/object_detection';
55

66
interface Props {
77
modelSource: string | number;
88
}
99

10-
interface ObjectDetectionModule {
10+
export const useObjectDetection = ({
11+
modelSource,
12+
}: Props): {
1113
error: string | null;
1214
isReady: boolean;
1315
isGenerating: boolean;
1416
forward: (input: string) => Promise<Detection[]>;
15-
}
16-
17-
export const useObjectDetection = ({
18-
modelSource,
19-
}: Props): ObjectDetectionModule => {
17+
} => {
2018
const [module, _] = useState(() => new _ObjectDetectionModule());
2119
const {
2220
error,

src/models/StyleTransfer.ts src/hooks/computer_vision/useStyleTransfer.ts

+6-8
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,19 @@
11
import { useState } from 'react';
2-
import { _StyleTransferModule } from '../native/RnExecutorchModules';
3-
import { useModule } from '../useModule';
2+
import { _StyleTransferModule } from '../../native/RnExecutorchModules';
3+
import { useModule } from '../../useModule';
44

55
interface Props {
66
modelSource: string | number;
77
}
88

9-
interface StyleTransferModule {
9+
export const useStyleTransfer = ({
10+
modelSource,
11+
}: Props): {
1012
error: string | null;
1113
isReady: boolean;
1214
isGenerating: boolean;
1315
forward: (input: string) => Promise<string>;
14-
}
15-
16-
export const useStyleTransfer = ({
17-
modelSource,
18-
}: Props): StyleTransferModule => {
16+
} => {
1917
const [module, _] = useState(() => new _StyleTransferModule());
2018
const {
2119
error,

src/ETModule.ts src/hooks/general/useExecutorchModule.ts

+12-5
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,23 @@
11
import { useState } from 'react';
2-
import { _ETModule } from './native/RnExecutorchModules';
3-
import { getError } from './Error';
4-
import { ExecutorchModule } from './types/common';
5-
import { useModule } from './useModule';
2+
import { _ETModule } from '../../native/RnExecutorchModules';
3+
import { useModule } from '../../useModule';
4+
import { ETInput } from '../../types/common';
5+
import { getError } from '../../Error';
66

77
interface Props {
88
modelSource: string | number;
99
}
1010

1111
export const useExecutorchModule = ({
1212
modelSource,
13-
}: Props): ExecutorchModule => {
13+
}: Props): {
14+
error: string | null;
15+
isReady: boolean;
16+
isGenerating: boolean;
17+
forward: (input: ETInput, shape: number[]) => Promise<number[][]>;
18+
loadMethod: (methodName: string) => Promise<void>;
19+
loadForward: () => Promise<void>;
20+
} => {
1421
const [module] = useState(() => new _ETModule());
1522
const {
1623
error,

src/LLM.ts src/hooks/natural_language_processing/useLLM.ts

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
import { useCallback, useEffect, useRef, useState } from 'react';
22
import { EventSubscription, Image } from 'react-native';
3-
import { ResourceSource, Model } from './types/common';
3+
import { ResourceSource, Model } from '../../types/common';
44
import {
55
DEFAULT_CONTEXT_WINDOW_LENGTH,
66
DEFAULT_SYSTEM_PROMPT,
77
EOT_TOKEN,
8-
} from './constants/llamaDefaults';
9-
import { LLM } from './native/RnExecutorchModules';
8+
} from '../../constants/llamaDefaults';
9+
import { LLM } from '../../native/RnExecutorchModules';
1010

1111
const interrupt = () => {
1212
LLM.interrupt();

src/index.tsx

+22-6
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,23 @@
1-
export * from './ETModule';
2-
export * from './LLM';
3-
export * from './constants/modelUrls';
4-
export * from './models/Classification';
5-
export * from './models/ObjectDetection';
6-
export * from './models/StyleTransfer';
1+
// hooks
2+
export * from './hooks/computer_vision/useClassification';
3+
export * from './hooks/computer_vision/useObjectDetection';
4+
export * from './hooks/computer_vision/useStyleTransfer';
5+
6+
export * from './hooks/natural_language_processing/useLLM';
7+
8+
export * from './hooks/general/useExecutorchModule';
9+
10+
// modules
11+
export * from './modules/computer_vision/ClassificationModule';
12+
export * from './modules/computer_vision/ObjectDetectionModule';
13+
export * from './modules/computer_vision/StyleTransferModule';
14+
15+
export * from './modules/natural_language_processing/LLMModule';
16+
17+
export * from './modules/general/ExecutorchModule';
18+
19+
// types
720
export * from './types/object_detection';
21+
22+
// constants
23+
export * from './constants/modelUrls';

0 commit comments

Comments
 (0)