Skip to content

Commit aec95c6

Browse files
NorbertKlockiewiczchmjkbMateusz Kopciński
authored
docs: Documentation for style transfer and bindings (#57)
## Description Style transfer and bindings docs pages ### Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] Documentation update (improves or adds clarity to existing documentation) ### Tested on - [ ] iOS - [ ] Android ### Testing instructions <!-- Provide step-by-step instructions on how to test your changes. Include setup details if necessary. --> ### Screenshots <!-- Add screenshots here, if applicable --> ### Related issues <!-- Link related issues here using #issue-number --> ### Checklist - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have updated the documentation accordingly - [ ] My changes generate no new warnings ### Additional notes <!-- Include any additional information, assumptions, or context that reviewers might need to understand this PR. --> --------- Co-authored-by: chmjkb <[email protected]> Co-authored-by: Mateusz Kopciński <[email protected]>
1 parent bbf6aee commit aec95c6

11 files changed

+375
-45
lines changed

‎docs/docs/computer-vision/_category_.json

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"label": "Computer Vision",
3+
"position": 3,
4+
"link": {
5+
"type": "generated-index"
6+
}
7+
}

‎docs/docs/computer-vision/useObjectDetection.mdx

+100
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
---
2+
title: useObjectDetection
3+
sidebar_position: 1
4+
---
5+
6+
`useObjectDetection` is a hook that lets you seamlessly integrate object detection into your React Native application. Currently, the SSDLite320Large model with MobileNetv3 backbone is supported.
7+
8+
## Reference
9+
```jsx
10+
import { useObjectDetection } from 'react-native-executorch';
11+
12+
function App() {
13+
const ssdlite = useObjectDetection({
14+
modelSource: require("./assets/ssdlite320large_mobilenetv3.pte"),
15+
});
16+
17+
...
18+
for (const detection of await ssdlite.forward("https://url-to-image.jpg")) {
19+
console.log("Bounding box: ", detection.bbox);
20+
console.log("Bounding label: ", detection.label);
21+
console.log("Bounding score: ", detection.score);
22+
}
23+
...
24+
}
25+
```
26+
27+
<details>
28+
<summary>Type definitions</summary>
29+
30+
```typescript
31+
```
32+
</details>
33+
34+
### Arguments
35+
36+
`modelSource`
37+
38+
A String that specifies the path to the model file. You can download the model from our HuggingFace repository.
39+
For SSDLite, you can add it to your assets directory, and use `require()`. If you prefer to download the model
40+
the model in runtime instead of bundling it, you can use the constants that we ship with the library.
41+
42+
### Returns
43+
44+
The hook returns an object with the following properties:
45+
46+
47+
| Field | Type | Description |
48+
| ------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------- |
49+
| `forward` | `(input: string) => Promise<Detection[]>` | Function that accepts an image and returns an array of Detection objects |
50+
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load or failed during generation process |
51+
| `isGenerating` | `boolean` | Indicates whether the model is processing the response |
52+
| `isReady` | `boolean` | Indicates whether the model has properly loaded and is ready for inference |
53+
54+
### Detection object
55+
The detection object is specified as follows:
56+
```typescript
57+
interface Bbox {
58+
x1: number;
59+
y1: number;
60+
x2: number;
61+
y2: number;
62+
}
63+
64+
interface Detection {
65+
bbox: Bbox;
66+
label: keyof typeof CocoLabels;
67+
score: number;
68+
}
69+
```
70+
The `bbox` property contains information about the bounding box of detected objects. It is represented as two points, one on the left bottom part of the bounding box (x1, y1), the second one as the top right part (x2, y2).
71+
The label property contains the name of the detected object, which is one of `CocoLabels`. The `score` is a confidence score of the detected object.
72+
73+
### Running the model
74+
75+
To run the model, you can use the `forward` method. It accepts one argument, which is the image. It can be either a remote URL,
76+
a local file or base64 encoded image. The function returns an array of `Detection` objects. Each one contains coordinates
77+
of the bounding box, the label of the detected object and confidence score. For more information, please refer to the reference or example.
78+
79+
### End to end example
80+
```tsx
81+
import { useObjectDetection, SSDLITE320LARGE_MOBILENETV3_WEIGHTS } from 'react-native-executorch';
82+
83+
function App() {
84+
const ssdlite = useObjectDetection({
85+
modelSource: SSDLITE320LARGE_MOBILENETV3_WEIGHTS, // Can also use require('') as well
86+
});
87+
88+
const runModel = async () => {
89+
const detections = await ssdlite.forward("https://url-to-image.jpg");
90+
for (const detection of detections) {
91+
console.log("Bounding box: ", detection.bbox); // [x, y, width, height]
92+
console.log("Bounding label: ", detection.label);
93+
console.log("Bounding score: ", detection.score);
94+
}
95+
}
96+
}
97+
```
98+
99+
### Benchmarks
100+
TODO

‎docs/docs/computer-vision/useStyleTransfer.md

+94
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
---
2+
title: useStyleTransfer
3+
sidebar_position: 1
4+
---
5+
6+
Style transfer is a technique used in computer graphics and machine learning where the visual style of one image is applied to the content of another. This is achieved using algorithms that manipulate data from both images, typically with the aid of a neural network. The result is a new image that combines the artistic elements of one picture with the structural details of another, effectively merging art with traditional imagery. React Native ExecuTorch offers a dedicated hook `useStyleTransfer`, for this task. However before you start you'll need to obtain ExecuTorch-compatible model binary.
7+
8+
:::caution
9+
It is recommended to use models provided by us which are available at our [HuggingFace repository](https://huggingface.co/software-mansion/react-native-executorch-style-transfer-candy), you can also use [constants](https://github.com/software-mansion/react-native-executorch/tree/main/src/constants/modelUrls.ts) shipped with our library
10+
:::
11+
12+
## Reference
13+
14+
```typescript
15+
import {
16+
useStyleTransfer,
17+
STYLE_TRANSFER_CANDY,
18+
} from 'react-native-executorch';
19+
20+
const model = useStyleTransfer({
21+
modelSource: STYLE_TRANSFER_CANDY,
22+
});
23+
24+
const imageUri = 'file::///Users/.../cute_cat.png';
25+
26+
try {
27+
const generatedImageUrl = await model.forward(imageUri);
28+
} catch (error) {
29+
console.error(error);
30+
}
31+
```
32+
33+
<details>
34+
<summary>Type definitions</summary>
35+
36+
```typescript
37+
interface StyleTransferModule {
38+
error: string | null;
39+
isReady: boolean;
40+
isGenerating: boolean;
41+
forward: (input: string) => Promise<string>;
42+
}
43+
```
44+
45+
</details>
46+
47+
### Arguments
48+
49+
**`modelSource`**
50+
A string that specifies the location of the model binary. For more information, take a look at [loading models](../fundamentals/loading-models.md) page.
51+
52+
### Returns
53+
54+
| Field | Type | Description |
55+
| -------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------- |
56+
| `forward` | `(input: string) => Promise<string>` | Executes the model's forward pass, where `input` can be a fetchable resource or a Base64-encoded string. |
57+
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load. |
58+
| `isGenerating` | `boolean` | Indicates whether the model is currently processing an inference. |
59+
| `isReady` | `boolean` | Indicates whether the model has successfully loaded and is ready for inference. |
60+
61+
## Running the model
62+
63+
To run the moel, you can use `forward` method. It accepts one argument, which is the image. The image can be a remote URL, a local file URI, or a base64-encoded image. The function returns a promise which can resolve either to error or a URL to generated image.
64+
65+
:::info
66+
Images from external sources and the generated image are stored in your application's temporary directory.
67+
:::
68+
69+
## Example
70+
71+
```typescript
72+
function App(){
73+
const model = useStyleTransfer(
74+
modelSource: STYLE_TRANSFER_CANDY,
75+
);
76+
77+
...
78+
const imageUri = 'file::///Users/.../cute_cat.png';
79+
80+
try{
81+
const generatedImageUrl = await model.forward(imageUri)
82+
}catch(error){
83+
console.error(error)
84+
}
85+
...
86+
}
87+
```
88+
89+
## Supported Models
90+
91+
- [Candy](https://github.com/pytorch/examples/tree/main/fast_neural_style)
92+
- [Mosaic](https://github.com/pytorch/examples/tree/main/fast_neural_style)
93+
- [Udnie](https://github.com/pytorch/examples/tree/main/fast_neural_style)
94+
- [Rain princess](https://github.com/pytorch/examples/tree/main/fast_neural_style)

‎docs/docs/fundamentals/getting-started.mdx

+2-2
Original file line numberDiff line numberDiff line change
@@ -45,11 +45,11 @@ If you plan on adding your models to the assets instead of fetching them from a
4545

4646
This allows us to use binaries, such as exported models or tokenizers for LLMs.
4747

48-
:::caution[Caution]
48+
:::caution
4949
When using Expo, please note that you need to use a custom development build of your app, not the standard Expo Go app. This is because we rely on native modules, which Expo Go doesn’t support.
5050
:::
5151

52-
:::info[Info]
52+
:::info
5353
Because we are using ExecuTorch under the hood, you won't be able to build ios app for release with simulator selected as the target device. Make sure to test release builds on real devices.
5454
:::
5555

‎docs/docs/fundamentals/loading-models.md

+45
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
---
2+
title: Loading models
3+
sidebar_position: 1
4+
---
5+
6+
There are three different methods available for loading model files, depending on their size and location.
7+
8+
**1. Load from React-Native assets folder (For Files < **512MB**)**
9+
10+
```typescript
11+
modelSource: require('../assets/llama3_2.pte');
12+
```
13+
14+
**2. Load from Remote URL:**
15+
16+
For files larger than 512MB or when you want to keep size of the app smaller, you can load the model from a remote URL (e.g. HuggingFace).
17+
18+
```typescript
19+
modelSource: 'https://.../llama3_2.pte';
20+
```
21+
22+
**3. Load from local file system:**
23+
24+
If you prefer to delegate the process of obtaining and loading model and tokenizer files to the user, you can use the following method:
25+
26+
```typescript
27+
modelSource: 'file::///var/mobile/.../llama3_2.pte',
28+
```
29+
30+
:::info
31+
The downloaded files are stored in documents directory of your application.
32+
:::
33+
34+
## Example
35+
36+
The following code snippet demonstrates how to load model and tokenizer files using `useLLM` hook:
37+
38+
```typescript
39+
import { useLLM } from 'react-native-executorch';
40+
41+
const llama = useLLM({
42+
modelSource: 'https://.../llama3_2.pte',
43+
tokenizer: require('../assets/tokenizer.bin'),
44+
});
45+
```

‎docs/docs/guides/_category_.json ‎docs/docs/llms/_category_.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"label": "Guides",
2+
"label": "LLMs",
33
"position": 2,
44
"link": {
55
"type": "generated-index"

‎docs/docs/guides/exporting-llama.mdx ‎docs/docs/llms/exporting-llama.mdx

File renamed without changes.

‎docs/docs/guides/running-llms.md ‎docs/docs/llms/running-llms.md

+4-34
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ sidebar_position: 1
66
React Native ExecuTorch supports Llama 3.2 models, including quantized versions. Before getting started, you’ll need to obtain the .pte binary—a serialized model—and the tokenizer. There are various ways to accomplish this:
77

88
- For your convienience, it's best if you use models exported by us, you can get them from our hugging face repository. You can also use [constants](https://github.com/software-mansion/react-native-executorch/tree/main/src/constants/modelUrls.ts) shipped with our library.
9-
- If you want to export model by yourself,you can use a Docker image that we've prepared. To see how it works, check out [exporting Llama](./exporting-llama.mdx)
9+
- If you want to export model by yourself,you can use a Docker image that we've prepared. To see how it works, check out [exporting Llama](./exporting-llama)
1010
- Follow the official [tutorial](https://github.com/pytorch/executorch/blob/fe20be98c/examples/demo-apps/android/LlamaDemo/docs/delegates/xnnpack_README.md) made by ExecuTorch team to build the model and tokenizer yourself
1111

1212
## Initializing
@@ -25,17 +25,17 @@ const llama = useLLM({
2525

2626
The code snippet above fetches the model from the specified URL, loads it into memory, and returns an object with various methods and properties for controlling the model. You can monitor the loading progress by checking the `llama.downloadProgress` and `llama.isReady` property, and if anything goes wrong, the `llama.error` property will contain the error message.
2727

28-
:::danger[Danger]
28+
:::danger
2929
Lower-end devices might not be able to fit LLMs into memory. We recommend using quantized models to reduce the memory footprint.
3030
:::
3131

32-
:::caution[Caution]
32+
:::caution
3333
Given computational constraints, our architecture is designed to support only one instance of the model runner at the time. Consequently, this means you can have only one active component leveraging `useLLM` concurrently.
3434
:::
3535

3636
### Arguments
3737

38-
**`modelSource`** - A string that specifies the location of the model binary. For more information, take a look at [loading models](#loading-models) section.
38+
**`modelSource`** - A string that specifies the location of the model binary. For more information, take a look at [loading models](../fundamentals/loading-models.md) section.
3939

4040
**`tokenizer`** - URL to the binary file which contains the tokenizer
4141

@@ -55,36 +55,6 @@ Given computational constraints, our architecture is designed to support only on
5555
| `isReady` | `boolean` | Indicates whether the model is ready |
5656
| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1, indicating the extent of the model file retrieval. |
5757

58-
### Loading models
59-
60-
There are three different methods available for loading the model and tokenizer files, depending on their size and location.
61-
62-
**1. Load from React-Native assets folder (For Files < **512MB**)**
63-
64-
```typescript
65-
modelSource: require('../assets/llama3_2.pte');
66-
```
67-
68-
**2. Load from Remote URL:**
69-
70-
For files larger than 512MB or when you want to keep size of the app smaller, you can load the model from a remote URL (e.g. HuggingFace).
71-
72-
```typescript
73-
modelSource: 'https://.../llama3_2.pte';
74-
```
75-
76-
**3. Load from local file system:**
77-
78-
If you prefer to delegate the process of obtaining and loading model and tokenizer files to the user, you can use the following method:
79-
80-
```typescript
81-
modelSource: 'file:://var/mobile/.../llama3_2.pte',
82-
```
83-
84-
:::info[Info]
85-
The downloaded files are stored in documents directory of your application.
86-
:::
87-
8858
### Sending a message
8959

9060
In order to send a message to the model, one can use the following code:

‎docs/docs/module-api/_category_.json

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"label": "Module API",
3+
"position": 4,
4+
"link": {
5+
"type": "generated-index"
6+
}
7+
}

0 commit comments

Comments
 (0)