Skip to content

Commit 8aa42fa

Browse files
jeffreylovitzjeffreylovitz
committed
Merge branch 'packaging' of https://github.com/filipecosta90/redisgraph-bulk-loader into filipecosta90-packaging
2 parents 0797051 + 69bdcdd commit 8aa42fa

File tree

6 files changed

+234
-8
lines changed

6 files changed

+234
-8
lines changed

‎.circleci/config.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ jobs:
2727
command: |
2828
python3 -m venv venv
2929
. venv/bin/activate
30-
pip install -r requirements.txt
30+
pip install .
3131
3232
- save_cache:
3333
paths:
@@ -38,8 +38,8 @@ jobs:
3838
name: examples
3939
command: |
4040
. venv/bin/activate
41-
python3 bulk_insert.py -n ./example/Person.csv -n ./example/Country.csv -r ./example/KNOWS.csv -r ./example/VISITED.csv Visits
42-
python3 bulk_insert.py -n ./example2/Robots.csv Robots
41+
redisgraph-bulk-loader -n ./example/Person.csv -n ./example/Country.csv -r ./example/KNOWS.csv -r ./example/VISITED.csv Visits
42+
redisgraph-bulk-loader -n ./example2/Robots.csv Robots
4343
4444
workflows:
4545
version: 2

‎.gitignore

Lines changed: 207 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,207 @@
1-
.vscode
1+
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm
2+
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
3+
4+
# User-specific stuff
5+
.idea/**/workspace.xml
6+
.idea/**/tasks.xml
7+
.idea/**/usage.statistics.xml
8+
.idea/**/dictionaries
9+
.idea/**/shelf
10+
11+
# Generated files
12+
.idea/**/contentModel.xml
13+
14+
# Sensitive or high-churn files
15+
.idea/**/dataSources/
16+
.idea/**/dataSources.ids
17+
.idea/**/dataSources.local.xml
18+
.idea/**/sqlDataSources.xml
19+
.idea/**/dynamic.xml
20+
.idea/**/uiDesigner.xml
21+
.idea/**/dbnavigator.xml
22+
23+
# Gradle
24+
.idea/**/gradle.xml
25+
.idea/**/libraries
26+
.idea/**
27+
28+
# Gradle and Maven with auto-import
29+
# When using Gradle or Maven with auto-import, you should exclude module files,
30+
# since they will be recreated, and may cause churn. Uncomment if using
31+
# auto-import.
32+
# .idea/modules.xml
33+
# .idea/*.iml
34+
# .idea/modules
35+
# *.iml
36+
# *.ipr
37+
38+
# CMake
39+
cmake-build-*/
40+
41+
# Mongo Explorer plugin
42+
.idea/**/mongoSettings.xml
43+
44+
# File-based project format
45+
*.iws
46+
47+
# IntelliJ
48+
out/
49+
50+
# mpeltonen/sbt-idea plugin
51+
.idea_modules/
52+
53+
# JIRA plugin
54+
atlassian-ide-plugin.xml
55+
56+
# Cursive Clojure plugin
57+
.idea/replstate.xml
58+
59+
# Crashlytics plugin (for Android Studio and IntelliJ)
60+
com_crashlytics_export_strings.xml
61+
crashlytics.properties
62+
crashlytics-build.properties
63+
fabric.properties
64+
65+
# Editor-based Rest Client
66+
.idea/httpRequests
67+
68+
# Android studio 3.1+ serialized cache file
69+
.idea/caches/build_file_checksums.ser
70+
71+
72+
# VisualStudioCode
73+
.vscode/*
74+
!.vscode/settings.json
75+
!.vscode/tasks.json
76+
!.vscode/launch.json
77+
!.vscode/extensions.json
78+
*.code-workspace
79+
80+
# Byte-compiled / optimized / DLL files
81+
__pycache__/
82+
*.py[cod]
83+
*$py.class
84+
85+
# C extensions
86+
*.so
87+
88+
# Distribution / packaging
89+
.Python
90+
build/
91+
develop-eggs/
92+
dist/
93+
downloads/
94+
eggs/
95+
.eggs/
96+
lib/
97+
lib64/
98+
parts/
99+
sdist/
100+
var/
101+
wheels/
102+
pip-wheel-metadata/
103+
share/python-wheels/
104+
*.egg-info/
105+
.installed.cfg
106+
*.egg
107+
MANIFEST
108+
109+
# PyInstaller
110+
# Usually these files are written by a python script from a template
111+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
112+
*.manifest
113+
*.spec
114+
115+
# Installer logs
116+
pip-log.txt
117+
pip-delete-this-directory.txt
118+
119+
# Unit test / coverage reports
120+
htmlcov/
121+
.tox/
122+
.nox/
123+
.coverage
124+
.coverage.*
125+
.cache
126+
nosetests.xml
127+
coverage.xml
128+
*.cover
129+
*.py,cover
130+
.hypothesis/
131+
.pytest_cache/
132+
133+
# Translations
134+
*.mo
135+
*.pot
136+
137+
# Django stuff:
138+
*.log
139+
local_settings.py
140+
db.sqlite3
141+
db.sqlite3-journal
142+
143+
# Flask stuff:
144+
instance/
145+
.webassets-cache
146+
147+
# Scrapy stuff:
148+
.scrapy
149+
150+
# Sphinx documentation
151+
docs/_build/
152+
153+
# PyBuilder
154+
target/
155+
156+
# Jupyter Notebook
157+
.ipynb_checkpoints
158+
159+
# IPython
160+
profile_default/
161+
ipython_config.py
162+
163+
# pyenv
164+
.python-version
165+
166+
# pipenv
167+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
168+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
169+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
170+
# install all needed dependencies.
171+
#Pipfile.lock
172+
173+
# celery beat schedule file
174+
celerybeat-schedule
175+
176+
# SageMath parsed files
177+
*.sage.py
178+
179+
# Environments
180+
.env
181+
.venv
182+
env/
183+
venv/
184+
ENV/
185+
env.bak/
186+
venv.bak/
187+
188+
# Spyder project settings
189+
.spyderproject
190+
.spyproject
191+
192+
# Rope project settings
193+
.ropeproject
194+
195+
# mkdocs documentation
196+
/site
197+
198+
# mypy
199+
.mypy_cache/
200+
.dmypy.json
201+
dmypy.json
202+
203+
# Pyre type checker
204+
.pyre/
205+
206+
# Mac
207+
*.DS_Store

‎README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,11 @@ https://oss.redislabs.com/redisgraph/
1818
## Installation
1919
The bulk loader script's dependencies can be resolved using pip:
2020
```
21-
pip install --user -r requirements.txt
21+
pip install git+https://github.com/RedisGraph/redisgraph-bulk-loader.git@master
2222
```
2323

2424
## Usage
25-
bulk_insert.py GRAPHNAME [OPTIONS]
25+
redisgraph-bulk-insert GRAPHNAME [OPTIONS]
2626

2727
| Flags | Extended flags | Parameter |
2828
|---------|-----------------------|-----------------------------------------------------------------|
@@ -44,15 +44,15 @@ The nodes and relationship flags should be specified once per input file.
4444
The flags for `max-token-count`, `max-buffer-size`, and `max-token-size` should only be specified if the memory overhead of graph creation is too high. The bulk loader builds large graphs by sending binary tokens (each of which holds multiple nodes or relations) to Redis in batches. By lowering these limits from their defaults, the size of each transmission to Redis is lowered and fewer entities are held in memory, at the expense of a longer overall runtime.
4545

4646
```
47-
python bulk_insert.py GRAPH_DEMO -n example/Person.csv -n example/Country.csv -r example/KNOWS.csv -r example/VISITED.csv
47+
redisgraph-bulk-insert GRAPH_DEMO -n example/Person.csv -n example/Country.csv -r example/KNOWS.csv -r example/VISITED.csv
4848
```
4949
The label (for nodes) or relationship type (for relationships) is derived from the base name of the input CSV file. In this example, we'll construct two sets of nodes, labeled `Person` and `Country`, and two types of relationships - `KNOWS` and `VISITED`.
5050

5151
The default behaviour is to infer the type for each row based on the value of each row, which can cause type mismatch problem. For example if a string property contains string values of 'false', 'true' or numbers. To avoid this, use --field-types to explicitly set the type for each column in the csv.
5252
EG, to explicitly set to string.
5353

5454
```
55-
python3 bulk_insert.py ROBOTS -f '{"Robots" : [3]}' -q1 -n example2/Robots.csv
55+
redisgraph-bulk-insert ROBOTS -f '{"Robots" : [3]}' -q1 -n example2/Robots.csv
5656
```
5757

5858
Notice that when -f isn't used, the robot name "30165" would be inserted as a number rather than a string which causes problems in RedisGraph when searching.

‎redisgraph_bulk_loader/__init__.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
from redisgraph_bulk_loader import bulk_insert
2+
3+
__all__ = [
4+
'bulk_insert',
5+
]

‎bulk_insert.py renamed to ‎redisgraph_bulk_loader/bulk_insert.py

File renamed without changes.

‎setup.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
from setuptools import setup, find_packages
2+
setup(
3+
name='redisgraph-bulk-loader',
4+
python_requires='>=3',
5+
version='0.8.1',
6+
packages=find_packages(),
7+
install_requires=[
8+
'redis',
9+
'click'
10+
],
11+
entry_points='''
12+
[console_scripts]
13+
redisgraph-bulk-loader=redisgraph_bulk_loader.bulk_insert:bulk_insert
14+
'''
15+
)

0 commit comments

Comments
 (0)