Build Executor (model) UI in Jina

[Nick17t]

Project idea 1: Build Executor (model) UI in jina

info	details
Skills needed	Python
Project size	175 hours
Difficulty level	Easy
Mentors	@Alaeddine Abdessalem, @Philip Vollet

Project Brief Description

• Jina Executors are components that perform certain tasks and expose them as services using gRPC. Executors accept DocumentArrays as input and output. However, with DocArray v2 focusing on type annotations and enabling annotation of executor endpoints, it becomes possible for executors to describe their services and input/output in the same way as OpenAPI schemas. This allows us to offer built-in UIs for executors, enabling people to easily use their services with multi-modal data. The goal is to build this feature in Jina using Gradio.

Detailed Description

A jina model UI should work like a swagger UI in an HTTP framework.
A swagger UI typically relies on an OpenAPI specifications to know about the services and input output schema, and then will generate the UI based on that.
However, jina executors rely on gRPC and gRPC do not expose OpenAPI. This means we either have to implement an OpenAPI equivalent for gRPC services that Executors expose or adopt a certain standard (I like the kserve).
Building such a service specification depends on knowing exactly the needed input and output schemas.
Jina can now support defining input and output schemas of the endpoints using the docarray v2 in beta support.
The API looks like this actually: https://docs.jina.ai/concepts/executor/docarray-v2/.
This means that, assuming an executor is built using DocArray v2 in type annotations, we can generate service specifications in the same way FastAPI generates open API specifications based on pydantic type hints.

Then, the next step would be using this service specification endpoint to build a model UI.
This can be done using gradio.
However, since we cannot serve a UI on the executor's gRPC service, we can only host the UI on the gateway level.
jina gateways recently added support for customization.
The new support allows basically building any server based on any protocol.
This means, a gateway can target an Executor within a jina flow and expose the UI for it.

I think the following API can make sense:

from jina import Flow

flow = Flow(protocol='ui').add(MyExecutorModel)

or

from jina import Flow, ModelUIGateway

flow = Flow().config_gateway(uses=ModelUIGateway).add(MyExecutorModel)

In jina, this would request the Executor's service specifications over grpc and then will dynamically build the gradio interface.

To sum it up, the proposed technologies are not necessarily what we need to adopt, as long as we implement the following:

• Executor service specifications (like OpenAPI specifications in http) based on DocArray v2 type hints
• Leverage the service specifications to build a model UI gateway. The UI Gateway is a custom Jina gateway and can use any framework (gradio, streamlit or even any web framework).
• Integrate the model ui gateway into jina (offer simple API to use it)

Expected outcomes

• Submit one or more Pull Requests (PRs) to the Jina repository that enables providing a built-in Executor UI for Executors. The UI can be built using Gradio and should be able to infer information about the Executor service using type annotations.

[Asrst]

Hi @Nick17t,

I would like to contribute to this as part of GSoC 2023. Just trying to understand the plan here - are we looking to automatically build some thing similar to swagger ui for each jina executor (but using gradio)?

Hi @alaeddine-13 , Any inputs here will help me to come up with a better proposal. pls let me know

[ranjan2829]

Project idea 1: Build Executor (model) UI in jina

info	details
Skills needed	Python
Project size	175 hours
Difficulty level	Easy
Mentors	@alaeddine Abdessalem, @philip Vollet

Project Description

• Jina Executors are components that perform certain tasks and expose them as services using gRPC. Executors accept DocumentArrays as input and output. However, with DocArray v2 focusing on type annotations and enabling annotation of executor endpoints, it becomes possible for executors to describe their services and input/output in the same way as OpenAPI schemas. This allows us to offer built-in UIs for executors, enabling people to easily use their services with multi-modal data. The goal is to build this feature in Jina using Gradio.

Expected outcomes

• Submit one or more Pull Requests (PRs) to the Jina repository that enables providing a built-in Executor UI for Executors. The UI can be built using Gradio and should be able to infer information about the Executor service using type annotations.

Project idea 1: Build Executor (model) UI in jina info details Skills needed Python Project size 175 hours Difficulty level Easy Mentors @[Alaeddine Abdessalem](https://github.com/alaeddine-13), @[Philip Vollet](https://www.linkedin.com/in/philipvollet) Project Description Jina Executors are components that perform certain tasks and expose them as services using gRPC. Executors accept DocumentArrays as input and output. However, with DocArray v2 focusing on type annotations and enabling annotation of executor endpoints, it becomes possible for executors to describe their services and input/output in the same way as OpenAPI schemas. This allows us to offer built-in UIs for executors, enabling people to easily use their services with multi-modal data. The goal is to build this feature in Jina using Gradio. Expected outcomes Submit one or more Pull Requests (PRs) to the Jina repository that enables providing a built-in Executor UI for Executors. The UI can be built using Gradio and should be able to infer information about the Executor service using type annotations.

Install Jina: First, you need to install Jina. You can use pip to install it:

Copy code
pip install jina

Define your Executor: Next, you need to define your Executor. For example, you can define a simple Executor that takes a list of strings as input and returns the length of each string:

from jina.executors import BaseExecutor

class MyExecutor(BaseExecutor):

def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)

def exec_fn(self, input):
    return [len(s) for s in input]

This code defines a MyExecutor class that inherits from Jina's BaseExecutor class. The MyExecutor class has an exec_fn method that takes a list of strings as input and returns the length of each string as a list.

Define a Flow: Once you have defined your Executor, you need to define a Flow that uses your Executor. For example, you can define a Flow that takes input from a REST API endpoint and passes it to your Executor:

from jina.flow import Flow

flow = Flow().add(
name='my_executor',
uses=MyExecutor,
parallel=2
)
This code defines a Flow object that uses your MyExecutor class. The parallel parameter specifies the number of parallel instances of the Executor to run.

Define a REST API endpoint: Next, you need to define a REST API endpoint that accepts input data and passes it to your Flow. For example, you can define a Flask app with a route that accepts POST requests:

python

_**from flask import Flask, request, jsonify

app = Flask(name)**_

@app.route('/api/execute', methods=['POST'])
def execute():
input_data = request.json['data']
with flow:
result = flow.index(input_data)
return jsonify({'result': result})
This code defines a Flask app with a /api/execute route that accepts POST requests. The route extracts the input data from the request, passes it to the Flow, and returns the result as a JSON response.

Run the Flask app: Finally, you need to run the Flask app to start the REST API endpoint:

if name == 'main':
app.run(debug=True, port=5000)
This code runs the Flask app with debugging enabled on port 5000.

That's it! With these steps, you should be able to build an Executor UI in Jina that uses a REST API endpoint to pass input data to your Executor and return the result. Of course, you will need to add more code to handle things like input validation, error handling, and authentication, but this should give you a good starting point.

[alaeddine-13]

Hey @Asrst @ranjan2829 ,
Thanks for your interest in contributing !

I believe your understanding is correct. A jina model UI should work like a swagger UI in an HTTP framework.
A swagger UI typically relies on an OpenAPI specifications to know about the services and input output schema, and then will generate the UI based on that.
However, jina executors rely on gRPC and gRPC do not expose OpenAPI. This means we either have to implement an OpenAPI equivalent for gRPC services that Executors expose or adopt a certain standard (I like the kserve).
Building such a service specification depends on knowing exactly the needed input and output schemas.
Jina can now support defining input and output schemas of the endpoints using the docarray v2 in beta support.
The API looks like this actually: https://docs.jina.ai/concepts/executor/docarray-v2/.
This means that, assuming an executor is built using DocArray v2 in type annotations, we can generate service specifications in the same way FastAPI generates open API specifications based on pydantic type hints.

Then, the next step would be using this service specification endpoint to build a model UI.
This can be done using gradio.
However, since we cannot serve a UI on the executor's gRPC service, we can only host the UI on the gateway level.
jina gateways recently added support for customization.
The new support allows basically building any server based on any protocol.
This means, a gateway can target an Executor within a jina flow and expose the UI for it.

I think the following API can make sense:

from jina import Flow

flow = Flow(protocol='ui').add(MyExecutorModel)

or

from jina import Flow, ModelUIGateway

flow = Flow().config_gateway(uses=ModelUIGateway).add(MyExecutorModel)

In jina, this would request the Executor's service specifications over grpc and then will dynamically build the gradio interface.

To sum it up, the proposed technologies are not necessarily what we need to adopt, as long as we implement the following:

• Executor service specifications (like OpenAPI specifications in http) based on DocArray v2 type hints
• Leverage the service specifications to build a model UI gateway. The UI Gateway is a custom Jina gateway and can use any framework (gradio, streamlit or even any web framework).
• Integrate the model ui gateway into jina (offer simple API to use it)

[ranjan2829]

@alaeddine-13
Thank you for providing more information about your feature request for a Jina model UI. Based on your explanation, it sounds like a very useful and valuable addition to the Jina platform.

I appreciate the level of detail you have provided about the technical implementation of the feature, as well as the proposed API. It's great to see that you have already thought through many of the technical challenges and possible solutions.

We will certainly consider your request and explore the feasibility of implementing it in a future release of Jina. As you mentioned, we may need to evaluate different technologies and frameworks to determine the best approach for implementing this feature, but we will definitely keep your proposed solution in mind.

Thank you again for your interest in contributing to the development of Jina. If you have any further suggestions or feedback, please do not hesitate to let us know.

Best regards,
Ranjan

[anurag2315]

from jina import Flow, Executor
from jina.types.document import DocumentArray
from jina.types.arrays.memmap import DocumentArrayMemmap
from jina.types.document.generators import from_ndarray
from jina import Document
from jina.types.ndarray.generic import NdArray
import gradio as gr

class MyExecutorModel(Executor):
def init(self, *args, **kwargs):
super().init(*args, **kwargs)

def foo(self, inputs: DocumentArray, *args, **kwargs) -> DocumentArray:
    for doc in inputs:
        # some processing logic
        doc.embedding = NdArray(np.random.rand(10,))
    return inputs

flow = Flow().add(MyExecutorModel)

Define the input and output schema using DocArray v2 type hints

input_schema = DocumentArrayMemmap(
field_resolver={
'text': 'string',
'image': 'blob'
}
)
output_schema = DocumentArrayMemmap(
field_resolver={
'embedding': 'ndarray'
}
)

Define the Gradio interface for the MyExecutorModel's "foo" method

def gradio_interface(inputs):
inputs = from_ndarray(inputs, input_schema)
outputs = flow.post(on='/foo', inputs=inputs, return_results=True)[0].data.docs
outputs = outputs.to_ndarray(output_schema)
return outputs

Define the Model UI Gateway and configure it to use the Gradio interface

from jina.types.request import Response
from jina import Gateway
from jina.proto.jina_pb2_grpc import GatewayServicer, add_GatewayServicer_to_server
from concurrent import futures
import grpc

class ModelUIGateway(Gateway):
class Service(GatewayServicer):
def init(self, gradio_interface):
self.gradio_interface = gradio_interface

    def Call(self, request, context):
        inputs = request.request
        outputs = self.gradio_interface(inputs)
        return Response(query=inputs, data=outputs)

def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    self.gradio_interface = None

def post_init(self):
    self.grpc_server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    add_GatewayServicer_to_server(self.Service(self.gradio_interface), self.grpc_server)
    self.grpc_server.add_insecure_port('[::]:50051')
    self.grpc_server.start()

def __call__(self, *args, **kwargs):
    self.gradio_interface = kwargs.pop('gradio_interface')
    super().__call__(*args, **kwargs)

Use the Model UI Gateway to host the Gradio interface for the MyExecutorModel's "foo" method

gradio_interface = gr.Interface(fn=gradio_interface, inputs=gr.inputs.Group([gr.inputs.Textbox(label="Text"), gr.inputs.Image(label="Image")]), outputs="numpy")
ui_gateway = ModelUIGateway().config(service_port=50051, uses='grpc://localhost:50051', uses_after='gradio_interface')
with flow, ui_gateway:
flow.protocol = 'ui'
flow.protocol_gateway = 'grpc://localhost:50051'
flow.protocol_gateway_args = {'gradio_interface': gradio_interface}
flow.block()

[AkbarHabeeb]

Dear Jina Team,
I'm interested in this idea. currently, I'm doing the feasibility analysis and designing various aspects of building an executor UI. Will share more details once ready.

[anurag2315]

@alaeddine-13 @philipvollet

i want to contribute in this project looking forward to work with you all.

[ranjan2829]

Project idea 1: Build Executor (model) UI in jina

info details
Skills needed Python
Project size 175 hours
Difficulty level Easy
Mentors @alaeddine Abdessalem, @philip Vollet

Project Description

• Jina Executors are components that perform certain tasks and expose them as services using gRPC. Executors accept DocumentArrays as input and output. However, with DocArray v2 focusing on type annotations and enabling annotation of executor endpoints, it becomes possible for executors to describe their services and input/output in the same way as OpenAPI schemas. This allows us to offer built-in UIs for executors, enabling people to easily use their services with multi-modal data. The goal is to build this feature in Jina using Gradio.

Expected outcomes

• Submit one or more Pull Requests (PRs) to the Jina repository that enables providing a built-in Executor UI for Executors. The UI can be built using Gradio and should be able to infer information about the Executor service using type annotations.

Project idea 1: Build Executor (model) UI in jina info details Skills needed Python Project size 175 hours Difficulty level Easy Mentors @Alaeddine Abdessalem, @Philip Vollet Project Description Jina Executors are components that perform certain tasks and expose them as services using gRPC. Executors accept DocumentArrays as input and output. However, with DocArray v2 focusing on type annotations and enabling annotation of executor endpoints, it becomes possible for executors to describe their services and input/output in the same way as OpenAPI schemas. This allows us to offer built-in UIs for executors, enabling people to easily use their services with multi-modal data. The goal is to build this feature in Jina using Gradio. Expected outcomes Submit one or more Pull Requests (PRs) to the Jina repository that enables providing a built-in Executor UI for Executors. The UI can be built using Gradio and should be able to infer information about the Executor service using type annotations.
Install Jina: First, you need to install Jina. You can use pip to install it:

Copy code pip install jina

Define your Executor: Next, you need to define your Executor. For example, you can define a simple Executor that takes a list of strings as input and returns the length of each string:

from jina.executors import BaseExecutor

class MyExecutor(BaseExecutor):

def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)

def exec_fn(self, input):
    return [len(s) for s in input]

This code defines a MyExecutor class that inherits from Jina's BaseExecutor class. The MyExecutor class has an exec_fn method that takes a list of strings as input and returns the length of each string as a list.

Define a Flow: Once you have defined your Executor, you need to define a Flow that uses your Executor. For example, you can define a Flow that takes input from a REST API endpoint and passes it to your Executor:

from jina.flow import Flow

flow = Flow().add( name='my_executor', uses=MyExecutor, parallel=2 ) This code defines a Flow object that uses your MyExecutor class. The parallel parameter specifies the number of parallel instances of the Executor to run.

Define a REST API endpoint: Next, you need to define a REST API endpoint that accepts input data and passes it to your Flow. For example, you can define a Flask app with a route that accepts POST requests:

python

_**from flask import Flask, request, jsonify

app = Flask(name)**_

@app.route('/api/execute', methods=['POST']) def execute(): input_data = request.json['data'] with flow: result = flow.index(input_data) return jsonify({'result': result}) This code defines a Flask app with a /api/execute route that accepts POST requests. The route extracts the input data from the request, passes it to the Flow, and returns the result as a JSON response.

Run the Flask app: Finally, you need to run the Flask app to start the REST API endpoint:

if name == 'main': app.run(debug=True, port=5000) This code runs the Flask app with debugging enabled on port 5000.

That's it! With these steps, you should be able to build an Executor UI in Jina that uses a REST API endpoint to pass input data to your Executor and return the result. Of course, you will need to add more code to handle things like input validation, error handling, and authentication, but this should give you a good starting point.

@alaeddine-13

[jamesbright]

I would love to be part of this project, looking forward to contribute and learn a lot.

[asuzukosi]

Here is a proof of concept for the implementation of the project. @alaeddine-13
@Nick17t what do you think? Also I would like to ask:

• if we implement a flow withe the UI gateway will that flow be used only of the UI of the executors?
• How do we account for the issue of flows having multiple executors?
• Will all the executors be listed on the same page? or will we have multiple urls for the same executor?
• If the UI protocol is used along with other protocols how will that affect the information displayed on the terminal

from pydantic import BaseModel
import gradio as gr

class InputModel(BaseModel):
    name: str
    id: int
    
class OutputModel(BaseModel):
    name: str
    othername: str
    

mappings = {
    str: gr.Textbox,
    int: gr.Number,
}

def get_gradio_interfaces(model):
    fields =  model.__fields__
    outputs = []
    for k,v in fields.items():
        intereface = mappings[v.type_]
        output = intereface(label=k)
        outputs.append(output)
    return outputs

print(get_gradio_interfaces(OutputModel))

def greet(*args, **kwargs):
    pass


def lauch_interfaces(inputModel, outputModel):
    demo = gr.Interface(
    fn=greet,
    inputs=get_gradio_interfaces(inputModel),
    outputs=get_gradio_interfaces(outputModel))
    demo.launch()

lauch_interfaces(inputModel=InputModel, outputModel=OutputModel)

[giuliocn]

Hello @ranjan2829,
I am considering this project for my application to GSoC 2023. It seems like I have the required Python skills, but I am new to open source contributions and I am wondering how many people (contributors) you will need to complete the project.

Thanks for the details and starting code you have shared above.

[iamharshvardhan]

Dear @ranjan2829,
I would like to contribute to the project for GSOC 2023. Although I have intermediate skills in python, I am eager to learn and contribute. I believe that this project will enable me to enhance my python skills and motivate me to do more open source contributions.

Thank you for considering.