feat: parallelization #352
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,10 @@ | ||||||||||
| export const kafkaConfig = { | ||||||||||
| clientId: 'maxun-scraper', | ||||||||||
| brokers: ['localhost:29092'], | ||||||||||
| topics: { | ||||||||||
|
Comment on lines
+1
to
+4
There was a problem hiding this comment. Use environment variables for Kafka configuration. The broker address and client ID are hardcoded, which limits deployment flexibility and poses security risks. Consider using environment variables for configuration. Apply this diff to make the configuration more flexible: +import { config } from 'dotenv';
+
+config();
+
export const kafkaConfig = {
- clientId: 'maxun-scraper',
- brokers: ['localhost:29092'],
+ clientId: process.env.KAFKA_CLIENT_ID || 'maxun-scraper',
+ brokers: (process.env.KAFKA_BROKERS || 'localhost:29092').split(','),Also, consider adding SSL/authentication configuration for production environments: // Add these properties to kafkaConfig
ssl: process.env.KAFKA_SSL === 'true',
sasl: process.env.KAFKA_SASL === 'true' ? {
mechanism: process.env.KAFKA_SASL_MECHANISM,
username: process.env.KAFKA_USERNAME,
password: process.env.KAFKA_PASSWORD,
} : undefined, |
||||||||||
| SCRAPING_TASKS: 'scraping-tasks', | ||||||||||
| SCRAPING_RESULTS: 'scraping-results', | ||||||||||
| SCRAPING_DLQ: 'scraping-dlq' | ||||||||||
| }, | ||||||||||
| consumerGroup: 'scraping-group' | ||||||||||
| }; | ||||||||||
|
Comment on lines
+9
to
+10
There was a problem hiding this comment. 🛠️ Refactor suggestion Add environment separation for consumer groups. The consumer group should be environment-aware to prevent cross-environment message processing. Apply this diff: - consumerGroup: 'scraping-group'
+ consumerGroup: `${process.env.ENV || 'dev'}-scraping-group`📝 Committable suggestion
Suggested change
Comment on lines
+1
to
+10
There was a problem hiding this comment. 💡 Codebase verification Remove unused Kafka configuration The Kafka configuration appears to be unused in the codebase. There are no imports of this configuration, no Kafka client instantiations, and no other Kafka-related code present. Consider removing this unused configuration to maintain a clean codebase. 🔗 Analysis chainVerify Kafka configuration in the codebase. Let's ensure the configuration is properly utilized across the codebase. 🏁 Scripts executedThe following scripts were executed for the analysis: Script: #!/bin/bash
# Description: Check for proper usage of Kafka configuration and potential security issues
# Check for hardcoded Kafka configurations
echo "Checking for hardcoded Kafka configurations..."
rg -g '!*.{json,md}' -i 'localhost:29092'
# Check for proper import usage
echo "Checking kafka config imports..."
rg "from ['|\"].*config/kafka" -A 3
# Check for potential security-sensitive Kafka operations
echo "Checking for direct Kafka operations..."
ast-grep --pattern 'new Kafka($$$)'
Length of output: 451 Script: #!/bin/bash
# Search for Kafka-related code and configurations
# Check for usage of kafkaConfig
echo "Checking for kafkaConfig usage..."
rg "kafkaConfig"
# Check for usage of topic names
echo "Checking for topic name usage..."
rg "scraping-tasks|scraping-results|scraping-dlq"
# Check for any Kafka-related imports or code
echo "Checking for Kafka-related code..."
rg -i "kafka" -g "!*.{json,md,txt}" -A 3
Length of output: 844 |
||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🛠️ Refactor suggestion
Pin the kafka-ui image version.
Using the
latesttag can lead to unexpected behavior when the image is updated. Pin to a specific version for reproducible builds.📝 Committable suggestion