Welcome_To_JavaWorld/KAFKA-Notes at main · polakiganeshkumar/Welcome_To_JavaWorld · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
streaming platform.


-- scalabilty -- heavy traffic tolerence
-- fault tolerance -- backups using brokers
-- durablility -- multiple consumers
-- realtime stream processing (Logs) -- order maintaining
-- pratitioning --
-- high throughput.
-- community and support.


for large amount of data in paticular order.
for loging your application logs at paticular place we use kafka.

ISR -- In sync replicas.

cosumer :
-- acknoledgemrnt
-- commit of offset.

High level :
+-----------------------------------------------------------------------+
|                           Kafka Cluster                               |
|                                                                       |
|  +-------------+       +------------+        +-------------+         |
|  |  Producer   | ----> |   Broker   | <----  |  Consumer   |         |
|  +-------------+       +------------+        +-------------+         |
|                                     |                                |
|  +-------------+                    |        +-------------+         |
|  |  Producer   | -------------------|----->  |  Consumer   |         |
|  +-------------+                    |        +-------------+         |
|                                     |                                |
|  +-------------+                    |                                |
|  |  Zookeeper  | <------------------|                                |
|  +-------------+       (Manages cluster metadata)                    |
+-----------------------------------------------------------------------+

Topis :
+---------------------+
|      Topic: "logs"  |
|  (Partitioned)      |
+---------------------+
| Partition 0 | Partition 1 | Partition 2 |
| (Leader:    | (Leader:    | (Leader:    |
|  Broker 1)  |  Broker 2)  |  Broker 3)  |
+-------------+-------------+-------------+
| Offset 0    | Offset 0    | Offset 0    |
| Offset 1    | Offset 1    | Offset 1    |
| ...         | ...         | ...         |
+----------------------------------------+

producer :
+-------------------+      +-------------------+      +-------------------+
|                   |      |                   |      |                   |
|  Producer App     | -->  |  Partitioner      | -->  |  Kafka Broker     |
|  (e.g., Web App)  |      | (Assigns to       |      |  (Leader for      |
|                   |      |  Partition)       |      |   Partition)      |
+-------------------+      +-------------------+      +-------------------+
                                      |
                                      v
                          (Optional: Acks=1/all for durability)


consumer :
+-------------------+      +-------------------+      +-------------------+
|                   |      |                   |      |                   |
|  Consumer Group   | -->  |  Partition        | -->  |  Kafka Broker     |
|  (e.g., "Group1") |      |  Assignment       |      |  (Reads from      |
|                   |      | (Balanced across  |      |   Partition)      |
|                   |      |  group members)   |      |                   |
+-------------------+      +-------------------+      +-------------------+
                                      |
                                      v
                          (Commits offsets to __consumer_offsets)