@@ -15,8 +15,24 @@ This project connects to the OpenSEA stream (non-REST) API to steam NFT transact
15
15
## Project Timeline
16
16
### Phase 1 - Project Setup
17
17
Setup of goals, basic project structure, timeline, virtual environments and version control.
18
+ 1 . Kafka, Spark, Jupyter Docker images setup, docker-compose setup and 1st git commit.
19
+ 2 . Node.js project directory setup and 1st git commit.
20
+ 3 . Java Gradle project directory setup and 1st git commit.
21
+ 4 . Azure account, git documentation creation and 1st git commit.
18
22
### Phase 2 - Duct Tape / Rapid Prototyping Phase (Current Phase)
19
23
Get all componented connected and the entire pipeline running in whatever form as quick as possible.
24
+ 1 . Kafka-Spark-Jupyter Docker stack: run containers. (done)
25
+ 2 . Kafka: Create test topics. (done)
26
+ 3 . Spark: Create and run spark session. (done)
27
+ 4 . Node.js app: Connect to OpenSEA API and produce json transaction console logs. (done)
28
+ 5 . Node.js app: Produce live json transaction data to kafka topics. (done)
29
+ 6 . Spark: Achieve consumption of kafka topic and basic transformation to another kafka topic. (done)
30
+ 7 . Azure Lakehouse: Achieve creation of Bronze, Silver and Gold containers. (done)
31
+ 8 . Azure Lakehouse: Achieve writing of simple delta file df data to lakehouse from local Spark Docker container. (done)
32
+ To-do:
33
+ 1 . Lakehouse-Tableau connection
34
+ 2 . Lakehouse-Plotly Dash connection
35
+ 3 . Kafka-Plotly Dash live connection
20
36
### Phase 3 - Feature Complete Phase
21
37
All main features implemented. Design Freeze.
22
38
### Phase 4 - Stabilizing Phase
0 commit comments