params.json

{"name":"Event-processing-platform","tagline":"Event processing platform backed by Spark","body":"# Reward Point App\r\n\r\n### Tech Stack\r\n* Java\r\n* Spark\r\n* Drools (Rule Engine)\r\n\r\n### Why I choose particularly this stack?\r\n* **Spark**  \r\n The most viable reason for choosing Spark instead of Multithreaded solution was scalability.\r\n i.e App should work for 1B+ transactions.  \r\n  \r\n  At first I thought to go with Multithreaded solution (Single Write Multi Read buffer) but that was clearly not scaling\r\n  for 1B+ transactions.  \r\n    \r\n    Then I thought of MapReduce. It can work with low memory and keeps part of files on disk. It brings them to memory\r\n    for merging (which is our case to accumulate reward points for each customer).  \r\n      \r\n    However **spark** provides best of both the worlds. It also uses memory and spills partitions to disk only when\r\n    required.\r\n* Drools (Rule Engine)  \r\n I first thought of implementing rule engine with `Chain of responsibility` design pattern. But what I wanted was\r\n something where in future if I have to make changes to existing rule or to a a new rule, I don't have to make additions\r\n in code. I wanted something that is controlled by some configuration. This save us from a lot of things like -\r\n code reviews, testing.  \r\n   \r\n   Hence I looked upon the internet for available rule engines and used Drools. It's a Bussiness Rule Management System.\r\n* Java  \r\n Major motivation to use java for implementing all this was my familiarity and experience with Java. \r\n\r\n\r\n### How is this App structured?\r\nMainly there are two packages which is of our interest.\r\n\r\n* core  \r\nIt contains classes that are required by any app like our current RewardPoint App or any such App that we may write in\r\nfuture that has same business use case like RewardPoint App.\r\n* rewardpoint\r\nIt contains classes specific to RewardPoint App which is this assignment.\r\n\r\n### How is this App modeled?\r\nAny App can be broken into three components/layers -\r\n* Input\r\n* Business Processing\r\n* Output\r\n\r\nI maintain these three sections in a configuration file, which an app reads to determine `input source (source),\r\noutput (sink)`. Config file for RewardPoint App is [RewardPointConfig.yaml](src/main/resources/RewardPointConfig.yaml)\r\n\r\nFew things about each section\r\n* Source  \r\nI have tried to keep source generic enough. Such that we can plugin any other source in future instead of CSV.\r\nWe can support HTTP, Kafka and others.\r\n\r\n* Sink  \r\nSame as source.\r\n\r\nCurrently Source and Sink are file based.  \r\nAlso we have some processing platform specific configs (in our case - spark) in config file.  \r\n\r\n### Class Diagrams\r\n* Config classe's structure  \r\n![Config classe's class diagram](src/main/resources/diagram/config-classes-class-diagram.jpg)  \r\nSo in here, RewardPointConfig is the config for our RewardPointApp.  \r\n  It extends AppConfig, which contains common configs for an App, like JobName, SourceConfig, SparkConfig.  \r\n    \r\n    \r\n* App classe's structure  \r\n![App classe's class diagram](src/main/resources/diagram/app-classes-class-diagram.jpg)\r\n\r\n\r\n### Sequence Diagram\r\n![Sequence diagram](src/main/resources/diagram/sequence-diagram.png)","note":"Don't delete this file! It's used internally to help with page regeneration."}