Welcome to Twitter Realtime Processing Repository

Python Realtime Processing Tweets COVID-19 using Kafka + Elasticsearch + Kibana + Docker + Docker-Compose

➤ Getting Started

If you want use this repository you need to make a git clone:

git clone --depth 1 https://github.com/lpmatos/twitter-realtime-processing-covid.git -b master

This will give access on your local machine.

➤ Pre-Requisites

To this project you yeed:

• Python 3.8.
• Docker and Docker Compose.
• Kafka ecosystem.
• Elasticsearch.
• Kiabana

➤ How to use it?

Locale

1. Set the application environment variables.
2. Install python packages in requirements.txt.
3. Run docker-compose.yml to deploy all kafka and elastic ecosystem.
4. Profit.

Docker

1. Set all environment variables in dot-env files.
2. Creathe a docker network.
3. Run docker-compose.yml to deploy all kafka and elastic ecosystem.
4. Run docker-compose-tools.to run the application.
5. Profit.

This system is fully containerised. You will need Docker and Docker Compose to run it.

You simply need to create a Docker network called kafka-network to enable communication between the Kafka cluster and the apps:

$ docker network create kafka-network

All set!

➤ Description

Sending Data to Elasticsearch

curl -X POST kafka-connect:8083/connectors -H "Content-Type: application/json" -d '{
	"name": "elasticsearch-sink-kafka",
	"config": {
		"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
		"type.name": "kafka-connect",
		"key.converter.schemas.enable": "false",
		"tasks.max": "1",
		"topics": "dados-tweets",
		"value.converter.schemas.enable": "false",
		"key.ignore": "true",
		"connection.url": " http://elasticsearch:9200",
		"value.converter": "org.apache.kafka.connect.json.JsonConverter",
		"key.converter": "org.apache.kafka.connect.storage.StringConverter",
		"schema.ignore": "true"
	}
}'

Environment variables

Name	Description
TWITTER_CONSUMER_KEY	Twitter Consumer Key
TWITTER_CONSUMER_SECRET	Twitter Consumer Secret
TWITTER_ACCESS_TOKEN	Twitter Access Token
TWITTER_ACCESS_TOKEN_SECRET	Twitter Access Token Secret
LOG_PATH	Just the Log Path
LOG_FILE	Just the Log File
LOG_LEVEL	Just the Log Level
LOGGER_NAME	Just the Logger name
KAFKA_BROKER_URL	Kafka Broker URL
KAFKA_TOPIC	Kafka Topic Name

Environment file

We use decouple for strict separation of settings from code. It helps us with to store parameters in .env file and properly convert values to correct data type.

Copy the file .env-example to a .env file and replace the values inside of it.

➤ Usage

Ways to run and use this project.

Docker

Steps to build the Docker Image.

Build

docker image build -t <IMAGE_NAME> -f <PATH_DOCKERFILE> <PATH_CONTEXT_DOCKERFILE>
docker image build -t <IMAGE_NAME> . (This context)

Run

Steps to run the Docker Container.

• Linux running:

docker container run -d -p <LOCAL_PORT:CONTAINER_PORT> <IMAGE_NAME> <COMMAND>
docker container run -it --rm --name <CONTAINER_NAME> -p <LOCAL_PORT:CONTAINER_PORT> <IMAGE_NAME> <COMMAND>

• Windows running:

winpty docker.exe container run -it --rm <IMAGE_NAME> <COMMAND>

For more information, access the Docker documentation or this.

Docker-Compose

Build and run a docker-compose.

docker-compose up --build

Down all services deployed by docker-compose.

docker-compose down

Down all services and delete all images.

docker-compose down --rmi all

➤ Visuals

Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.

➤ Author

👤 Lucca Pessoa

Hey!! If you like this project or if you find some bugs feel free to contact me in my channels:

• Email: lpsm-dev@protonmail.com
• Website: https://github.com/lpmatos
• GitHub: @lpmatos
• GitLab: @lpmatos

➤ Versioning

To check the change history, please access the CHANGELOG.md file.

➤ Troubleshooting

If you have any problems, please contact me.

➤ Project status

This project is currently undergoing a reorganization 👾.

➤ Show your support

Give me a ⭐️ if this project helped you!

Made with 💜 by me 👋 inspired on readme-md-generator