May anybody please share in the event that they designed a system, utilizing Kafka, whereas requiring large volumes of site visitors, and no messages are allowed to be misplaced?
E.g. say a “Procuring Website” will get orders and forwards them (by means of Kafka) to “Bill System” that points invoices. It is essential to not miss any invoices:
[Shopping Site] ---kafka--> [Invoices System]
I am new to Kafka, however heard it may lose messages even when utilizing cautious configuration (eg https://jack-vanlightly.com/weblog/2018/9/14/how-to-lose-messages-on-a-kafka-cluster-part1). That is comprehensible, however how do I compensate in my design?
With smaller techniques, I used databases: one database for “Procuring Website”, one other database for “Invoices”, and I may correlate them on some nightly course of, to catch any missed invoices. However I have not learn any such suggestions for Kafka, and I am additionally frightened it is going to be unrealistic to correlate the tables in the event that they’re large.
As anybody met comparable necessities and confronted an identical dilemma? Any methods that labored?