DBreplex
DBreplex is an Elixir application that enables replication of data from databases to (potentially) any target storage. It utilizes replication capabilities of modern database engines for push-based download of data and subsequent upload to target.
The diagram below shows a high level overview of the application architecture. Each process component in the diagram represents Elixir's process (Agent/GenServer/...). As can be seen the whole application is divided into two layers: Subscriber layer and Publisher layer.
Subscriber layer is responsible for establishing the replication connection to a database --source of data-- and handling of incoming messages.
In this context, the replication connection is a special type of database connection. It signals the source database that the process on the other end of the connection is ready to receive messages. Basically, it behaves as a subscriber client to the source (publishing) database.
Handling of messages consists of two steps: parse a message to a generic (core) format and send it to all publishers.
Publisher layer reacts to incoming messages by parsing them to target specific format and uploading them to target. Note, that publishers receive messages in core format. Core format of a message contains the least amount of information to be able to represent target specific messages.