Skip to content

Commit f8d2423

Browse files
MikeBirdTechMikeBirdTech
committed
Add "Open Interpreter and ScreenPipe Cookbook" IPython notebook with examples and code snippets for integrating Open Interpreter and ScreenPipe.
1 parent b4b9450 commit f8d2423

File tree

1 file changed

+220
-0
lines changed

1 file changed

+220
-0
lines changed

‎examples/screenpipe.ipynb

+220
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Open Interpreter and ScreenPipe Cookbook\n",
8+
"\n",
9+
"This cookbook explores the powerful combination of Open Interpreter and ScreenPipe, two tools that can significantly enhance your ability to interact with and process digital information. Open Interpreter allows you to execute natural language commands, while ScreenPipe captures and analyzes screen content and audio output from your computer."
10+
]
11+
},
12+
{
13+
"cell_type": "markdown",
14+
"metadata": {},
15+
"source": [
16+
"## Prerequisites\n",
17+
"\n",
18+
"Before we begin, make sure you have installed the following:\n",
19+
"\n",
20+
"1. [Open Interpreter](https://docs.openinterpreter.com/getting-started/introduction)\n",
21+
"2. [Screenpipe CLI](https://docs.screenpi.pe/docs/getting-started#cli-installation)\n",
22+
"\n",
23+
"Make sure both are properly installed and configured on your system."
24+
]
25+
},
26+
{
27+
"cell_type": "markdown",
28+
"metadata": {},
29+
"source": [
30+
"## Setting Up Open Interpreter"
31+
]
32+
},
33+
{
34+
"cell_type": "code",
35+
"execution_count": null,
36+
"metadata": {},
37+
"outputs": [],
38+
"source": [
39+
"# Import necessary libraries\n",
40+
"from interpreter import interpreter\n",
41+
"from datetime import datetime, timezone\n",
42+
"\n",
43+
"# Configure Open Interpreter\n",
44+
"interpreter.llm.model = \"groq/llama-3.1-70b-versatile\"\n",
45+
"interpreter.computer.import_computer_api = False\n",
46+
"interpreter.llm.supports_functions = False\n",
47+
"interpreter.llm.supports_vision = False\n",
48+
"interpreter.llm.context_window = 100000\n",
49+
"interpreter.llm.max_tokens = 4096\n",
50+
"\n",
51+
"# Add the current date and time in UTC\n",
52+
"current_datetime = datetime.now(timezone.utc).strftime(\"%Y-%m-%d %H:%M:%S UTC\")\n",
53+
"print(f\"Current date and time: {current_datetime}\")"
54+
]
55+
},
56+
{
57+
"cell_type": "markdown",
58+
"metadata": {},
59+
"source": [
60+
"## Defining the ScreenPipe Search Function"
61+
]
62+
},
63+
{
64+
"cell_type": "code",
65+
"execution_count": null,
66+
"metadata": {},
67+
"outputs": [],
68+
"source": [
69+
"# Define the custom ScreenPipe search function\n",
70+
"custom_tool = \"\"\"\n",
71+
"import requests\n",
72+
"import json\n",
73+
"from urllib.parse import quote\n",
74+
"\n",
75+
"def search_screenpipe(query, limit=5, start_time=None, end_time=None):\n",
76+
" base_url = f\"http://localhost:3030/search?q={quote(query)}&content_type=ocr&limit={limit}\"\n",
77+
" \n",
78+
" if start_time:\n",
79+
" base_url += f\"&start_time={quote(start_time)}\"\n",
80+
" if end_time:\n",
81+
" base_url += f\"&end_time={quote(end_time)}\"\n",
82+
" \n",
83+
" response = requests.get(base_url)\n",
84+
" if response.status_code == 200:\n",
85+
" data = response.json()\n",
86+
" # Remove duplicates based on text content\n",
87+
" unique_results = []\n",
88+
" seen_texts = set()\n",
89+
" for item in data[\"data\"]:\n",
90+
" text = item[\"content\"][\"text\"]\n",
91+
" if text not in seen_texts:\n",
92+
" unique_results.append(item)\n",
93+
" seen_texts.add(text)\n",
94+
" return unique_results\n",
95+
" else:\n",
96+
" return f\"Error: Unable to fetch data from ScreenPipe. Status code: {response.status_code}\"\n",
97+
"\"\"\"\n",
98+
"\n",
99+
"# Add the custom tool to the interpreter's environment\n",
100+
"interpreter.computer.run(\"python\", custom_tool)\n",
101+
"print(\"ScreenPipe search function defined and added to the interpreter's environment.\")"
102+
]
103+
},
104+
{
105+
"cell_type": "markdown",
106+
"metadata": {},
107+
"source": [
108+
"## Setting Custom Instructions for Open Interpreter"
109+
]
110+
},
111+
{
112+
"cell_type": "code",
113+
"execution_count": null,
114+
"metadata": {},
115+
"outputs": [],
116+
"source": [
117+
"# Set custom instructions for Open Interpreter\n",
118+
"interpreter.custom_instructions = f\"\"\"\n",
119+
"Current date and time: {current_datetime}\n",
120+
"\n",
121+
"ScreenPipe is a powerful tool that continuously captures and indexes the content displayed on your screen. It creates a searchable history of everything you've seen or interacted with on your computer. This includes text from websites, documents, applications, and even images (through OCR).\n",
122+
"\n",
123+
"You have access to this wealth of information through the `search_screenpipe(query, limit=5, start_time=None, end_time=None)` function. This allows you to provide more contextual and personalized assistance based on the user's recent activities and viewed content.\n",
124+
"\n",
125+
"The `search_screenpipe` function supports optional `start_time` and `end_time` parameters to narrow down the search to a specific time range. The time format should be ISO 8601, like this: \"2024-10-16T12:00:00Z\".\n",
126+
"\n",
127+
"Here's why querying ScreenPipe is valuable:\n",
128+
"1. Context Recall: Users often refer to things they've seen recently but may not remember the exact details. ScreenPipe can help recall this information.\n",
129+
"2. Information Verification: You can cross-reference user claims or questions with actual content they've viewed.\n",
130+
"3. Personalized Assistance: By knowing what the user has been working on or researching, you can provide more relevant advice and suggestions.\n",
131+
"4. Productivity Enhancement: You can help users quickly locate information they've seen before but can't remember where.\n",
132+
"\n",
133+
"Use the `search_screenpipe()` function when:\n",
134+
"- The user asks about something they've seen or read recently.\n",
135+
"- You need to verify or expand on information the user mentions.\n",
136+
"- You want to provide context-aware suggestions or assistance.\n",
137+
"- The user is trying to recall specific details from their recent computer usage.\n",
138+
"- The user wants to search within a specific time range.\n",
139+
"\n",
140+
"Here's how to use it effectively:\n",
141+
"1. When a user's query relates to recent activities or viewed content, identify key terms for the search.\n",
142+
"2. If the user specifies a time range, use the `start_time` and `end_time` parameters.\n",
143+
"3. Call the `search_screenpipe()` function with these parameters.\n",
144+
"4. Analyze the results to find relevant information.\n",
145+
"5. Summarize the findings for the user, mentioning the source (app name, window name) and when it was seen (timestamp).\n",
146+
"\n",
147+
"Remember to use this tool proactively when you think it might help answer the user's question, even if they don't explicitly mention ScreenPipe.\n",
148+
"\"\"\"\n",
149+
"\n",
150+
"print(\"Custom instructions set for Open Interpreter.\")"
151+
]
152+
},
153+
{
154+
"cell_type": "markdown",
155+
"metadata": {},
156+
"source": [
157+
"## Example Usage"
158+
]
159+
},
160+
{
161+
"cell_type": "code",
162+
"execution_count": null,
163+
"metadata": {},
164+
"outputs": [],
165+
"source": [
166+
"# Example usage of the search_screenpipe function\n",
167+
"\n",
168+
"# Search without time range\n",
169+
"print(\"Searching for 'Open Interpreter' without time range:\")\n",
170+
"results = search_screenpipe(\"Open Interpreter\", limit=3)\n",
171+
"for result in results:\n",
172+
" print(f\"Text: {result['content']['text'][:300]}...\")\n",
173+
" print(f\"Source: {result['content']['app_name']} - {result['content']['window_name']}\")\n",
174+
" print(f\"Timestamp: {result['content']['timestamp']}\")\n",
175+
" print()\n",
176+
"\n",
177+
"# Search with time range\n",
178+
"print(\"\\nSearching for 'project meeting' with time range:\")\n",
179+
"results = search_screenpipe(\"project meeting\", limit=5, start_time=\"2024-10-16T12:00:00Z\", end_time=\"2024-10-16T19:00:00Z\")\n",
180+
"for result in results:\n",
181+
" print(f\"Text: {result['content']['text'][:300]}...\")\n",
182+
" print(f\"Source: {result['content']['app_name']} - {result['content']['window_name']}\")\n",
183+
" print(f\"Timestamp: {result['content']['timestamp']}\")\n",
184+
" print()"
185+
]
186+
},
187+
{
188+
"cell_type": "markdown",
189+
"metadata": {},
190+
"source": [
191+
"## Conclusion\n",
192+
"\n",
193+
"This notebook demonstrates how to set up and use Open Interpreter in conjunction with ScreenPipe. By leveraging these tools together, you can create powerful, context-aware applications that can search and analyze your screen content history.\n",
194+
"\n",
195+
"Remember to adjust the search queries and time ranges in the example usage section to fit your specific use case. Happy coding!"
196+
]
197+
}
198+
],
199+
"metadata": {
200+
"kernelspec": {
201+
"display_name": "Python 3",
202+
"language": "python",
203+
"name": "python3"
204+
},
205+
"language_info": {
206+
"codemirror_mode": {
207+
"name": "ipython",
208+
"version": 3
209+
},
210+
"file_extension": ".py",
211+
"mimetype": "text/x-python",
212+
"name": "python",
213+
"nbconvert_exporter": "python",
214+
"pygments_lexer": "ipython3",
215+
"version": "3.8.10"
216+
}
217+
},
218+
"nbformat": 4,
219+
"nbformat_minor": 4
220+
}

0 commit comments

Comments
 (0)