Final Task - Sholomitski Dmitry

Final_Task.md

@@ -0,0 +1,195 @@
+# Introduction to Python. Final task.
+You are proposed to implement Python RSS-reader using  **python 3.9**.
+
+The task consists of few iterations. Do not start new iteration if the previous one is not implemented yet.
+
+## Common requirements.
+* It is mandatory to use `argparse` module.
+* Codebase must be covered with unit tests with at least 50% coverage. It's a mandatory requirement.
+* Yor script should **not** require installation of other services such as mysql server,
+postgresql and etc. (except Iteration 6). If it does require such programs,
+they should be installed automatically by your script, without user doing anything.
+* In case of any mistakes utility should print human-readable.
+error explanation. Exception tracebacks in stdout are prohibited in final version of application.
+* Docstrings are mandatory for all methods, classes, functions and modules.
+* Code must correspond to `pep8` (use `pycodestyle` utility for self-check).
+  * You can set line length up to 120 symbols.
+* Commit messages should provide correct and helpful information about changes in commit. Messages like `Fix bug`, 
+`Tried to make workable`, `Temp commit` and `Finally works` are prohibited.
+* All used third-party packages should be written in the `requirements.txt` file and in installation files (`setup.py`, `setup.cfg`, etc.).
+* You have to write a file with documentation. Everything must be documented: how to run scripts, how to run tests, how to install the library and etc.
+
+## [Iteration 1] One-shot command-line RSS reader.
+RSS reader should be a command-line utility which receives [RSS](wikipedia.org/wiki/RSS) URL and prints results in human-readable format.
+
+You are free to choose format of the news console output. The textbox below provides an example of how it can be implemented:
+
+```shell
+$ rss_reader.py "https://news.yahoo.com/rss/" --limit 1
+
+Feed: Yahoo News - Latest News & Headlines
+
+Title: Nestor heads into Georgia after tornados damage Florida
+Date: Sun, 20 Oct 2019 04:21:44 +0300
+Link: https://news.yahoo.com/wet-weekend-tropical-storm-warnings-131131925.html
+
+[image 2: Nestor heads into Georgia after tornados damage Florida][2]Nestor raced across Georgia as a post-tropical cyclone late Saturday, hours after the former tropical storm spawned a tornado that damaged
+homes and a school in central Florida while sparing areas of the Florida Panhandle devastated one year earlier by Hurricane Michael. The storm made landfall Saturday on St. Vincent Island, a nature preserve
+off Florida's northern Gulf Coast in a lightly populated area of the state, the National Hurricane Center said. Nestor was expected to bring 1 to 3 inches of rain to drought-stricken inland areas on its
+march across a swath of the U.S. Southeast.
+
+
+Links:
+[1]: https://news.yahoo.com/wet-weekend-tropical-storm-warnings-131131925.html (link)
+[2]: http://l2.yimg.com/uu/api/res/1.2/Liyq2kH4HqlYHaS5BmZWpw--/YXBwaWQ9eXRhY2h5b247aD04Njt3PTEzMDs-/https://media.zenfs.com/en/ap.org/5ecc06358726cabef94585f99050f4f0 (image)
+
+```
+
+Utility should provide the following interface:
+```shell
+usage: rss_reader.py [-h] [--version] [--json] [--verbose] [--limit LIMIT]
+                     source
+
+Pure Python command-line RSS reader.
+
+positional arguments:
+  source         RSS URL
+
+optional arguments:
+  -h, --help     show this help message and exit
+  --version      Print version info
+  --json         Print result as JSON in stdout
+  --verbose      Outputs verbose status messages
+  --limit LIMIT  Limit news topics if this parameter provided
+
+```
+
+In case of using `--json` argument your utility should convert the news into [JSON](https://en.wikipedia.org/wiki/JSON) format.
+You should come up with the JSON structure on you own and describe it in the README.md file for your repository or in a separate documentation file.
+
+
+
+With the argument `--verbose` your program should print all logs in stdout.
+
+### Task clarification (I)
+
+1) If `--version` option is specified app should _just print its version_ and stop.
+2) User should be able to use `--version` option without specifying RSS URL. For example:
+```
+> python rss_reader.py --version
+"Version 1.4"
+```
+3) The version is supposed to change with every iteration.
+4) If `--limit` is not specified, then user should get _all_ available feed.
+5) If `--limit` is larger than feed size then user should get _all_ available news.
+6) `--verbose` should print logs _in the process_ of application running, _not after everything is done_.
+7) Make sure that your app **has no encoding issues** (meaning symbols like `&#39` and etc) when printing news to _stdout_.
+8) Make sure that your app **has no encoding issues** (meaning symbols like `&#39` and etc) when printing news to _stdout in JSON format_.
+9) It is preferrable to have different custom exceptions for different situations(If needed).
+10) The `--limit` argument should also affect JSON generation.
+
+
+## [Iteration 2] Distribution.
+
+* Utility should be wrapped into distribution package with `setuptools`.
+* This package should export CLI utility named `rss-reader`.
+
+
+### Task clarification (II)
+
+1) User should be able to run your application _both_ with and without installation of CLI utility,
+meaning that this should work:
+
+```
+> python rss_reader.py ...
+```
+
+as well as this:  
+
+```
+> rss_reader ...
+```
+2) Make sure your second iteration works on a clean machie with python 3.9. (!)
+3) Keep in mind that installed CLI utility should have the same functionality, so do not forget to update dependencies and packages.
+
+
+## [Iteration 3] News caching.
+The RSS news should be stored in a local storage while reading. The way and format of this storage you can choose yourself.
+Please describe it in a separate section of README.md or in the documentation.
+
+New optional argument `--date` must be added to your utility. It should take a date in `%Y%m%d` format.
+For example: `--date 20191020`
+Here date means actual *publishing date* not the date when you fetched the news.
+
+The cashed news can be read with it. The new from the specified day will be printed out.
+If the news are not found return an error.
+
+If the `--date` argument is not provided, the utility should work like in the previous iterations.
+
+### Task clarification (III)
+1) Try to make your application crossplatform, meaning that it should work on both Linux and Windows.
+For example when working with filesystem, try to use `os.path` lib instead of manually concatenating file paths.
+2) `--date` should **not** require internet connection to fetch news from local cache.
+3) User should be able to use `--date` without specifying RSS source. For example:
+```
+> python rss_reader.py --date 20191206
+......
+```
+Or for second iteration (when installed using setuptools):
+```
+> rss_reader --date 20191206
+......
+```
+4) If `--date` specified _together with RSS source_, then app should get news _for this date_ from local cache that _were fetched from specified source_.
+5) `--date` should work correctly with both `--json`, `--limit`, `--verbose` and their different combinations.
+
+## [Iteration 4] Format converter.
+
+You should implement the conversion of news in at least two of the suggested format: `.mobi`, `.epub`, `.fb2`, `.html`, `.pdf`
+
+New optional argument must be added to your utility. This argument receives the path where new file will be saved. The arguments should represents which format will be generated.
+
+For example:  `--to-mobi` or `--to-fb2` or `--to-epub`
+
+You can choose yourself the way in which the news will be displayed, but the final text result should contain pictures and links, if they exist in the original article and if the  format permits to store this type of data.
+
+### Task clarification (IV)
+
+Convertation options should work correctly together with all arguments that were implemented in Iterations 1-3. For example: 
+* Format convertation process should be influenced by `--limit`.
+* If `--json` is specified together with convertation options, then JSON news should 
+be printed to stdout, and converted file should contain news in normal format.
+* Logs from `--verbose` should be printed in stdout and not added to the resulting file.
+* `--date` should also work correctly with format converter and to not require internet access.
+
+## * [Iteration 5] Output colorization.
+> Note: An optional iteration, it is not necessary to implement it. You can move on with it only if all the previous iterations (from 1 to 4) are completely implemented.
+
+You should add new optional argument `--colorize`, that will print the result of the utility in colorized mode.
+
+*If the argument is not provided, the utility should work like in the previous iterations.*
+
+> Note: Take a look at the [colorize](https://pypi.org/project/colorize/) library
+
+## * [Iteration 6] Web-server.
+> Note: An optional iteration, it is not necessary to implement it. You can move on with it only if all the previous iterations (from 1 to 4) are completely implemented. Introduction to Python course does not cover the topics that are needed for the implementation of this part.
+
+There are several mandatory requirements in this iteration:
+* `Docker` + `docker-compose` usage (at least 2 containers: one for web-application, one for DB)
+* Web application should provide all the implemented in the previous parts of the task functionality, using the REST API:
+    * One-shot conversion from RSS to Human readable format
+    * Server-side news caching
+    * Conversion in epub, mobi, fb2 or other formats
+
+Feel free to choose the way of implementation, libraries and frameworks. (We suggest you `Django Rest Framework` + `PostgreSQL` combination)
+
+You can implement any functionality that you want. The only requirement is to add the description into README file or update project documentation, for example:
+* authorization/authentication
+* automatic scheduled news update
+* adding new RSS sources using API
+
+---
+Implementations will be checked with the latest cPython interpreter of 3.9 branch.
+---
+
+> Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. Code for readability. **John F. Woods**

RULES.md

@@ -0,0 +1,12 @@
+# Final task
+Final task (`FT`) for EPAM Python Training 2022.03
+
+## Rules
+* All work has to be implemented in the `master` branch in forked repository. If you think that `FT` is ready, please open a pull request (`PR`) to our repo.
+* When a `PR` will be ready, please mark it with the `final_task` label.
+* You have one month to finish `FT`. Commits commited after deadline will be ignored.
+* At least the first 4 iterations must be done.
+* `FT` you can find in the `Final_Task.md` file.
+
+### Good luck!
+

Sholomitski Dmitry/MANIFEST.in

@@ -0,0 +1,2 @@
+include README.md
+include requirements.txt

Sholomitski Dmitry/requirements.txt

@@ -0,0 +1,9 @@
+setuptools~=62.1.0
+requests~=2.27.1
+bs4~=0.0.1
+beautifulsoup4~=4.11.1
+dateparser~=1.1.1
+reportlab~=3.6.10
+xhtml2pdf~=0.2.8
+colorama~=0.4.4
+Pygments~=2.12.0

Sholomitski Dmitry/rss_reader/README.md

@@ -0,0 +1,69 @@
+# Final Task Final Task for EPAM Python Training by Sholomitski Dmitry
+
+I am proposed to implement Python RSS-reader using python 3.10.
+
+RSS reader is a command-line utility which receives RSS URL and prints results in human-readable format. Format of the news console output: $ rss_reader.py https://people.onliner.by/feed
+
+```
+channel title:  Лайфстайл Onlíner
+channel link:  https://people.onliner.by/
+************************************************************************************************************************
+Title: В шестнадцать лет выиграла Олимпиаду, а в семнадцать — попалась на магазинной краже. Рассказываем о несчастливом вундеркинде женского тенниса (спецпроект)
+Description: Богатые тоже плачут. Даже если они успешны, знамениты и обожаемы сотнями тысяч поклонников. Это относится и к американской теннисистке Дженнифер Каприати, которую снабдила талантом сама природа. Еще в подростковом возрасте спортсменка начала выносить взрослых соперниц. Уже в девять лет юной Дженни предлагали сотрудничество производители теннисной формы и снаряжения. А ведущие американские СМИ стали наблюдать за карьерой Каприати задолго до того, как теннисистка окончила школу. Вместе с BETERA в нашем спортивном проекте рассказываем, как «золотая» девочка из спорта докатилась до магазинной кражи, марихуаны и мыслей о суициде.Читать далее…
+Published: 2022-06-30 16:11:31
+Image: https://content.onliner.by/news/thumbnail/19f511e02774f2c9d0c7d74e66342dbb.jpeg
+Read more: https://people.onliner.by/2022/06/30/vunderkind
+------------------------------------------------------------------------------------------------------------------------
+```
+
+The program has the following settings, which can be applied together or separately
+```
+-h, --help     show this help message and exit
+--version      Print version info and exit
+--json         Save result as in JSON file
+--limit LIMIT  Limit news if this parameter provided
+-v, --verbose  Output verbose messages
+--date DATE    Gets a date in YYYYMMDD format. Print news from the specified date.
+--to-html      Convert news to HTML file (provide path to folder where save export file)
+--to-pdf       Convert news to PDF file (provide path to folder where save export file)
+--colorize     Enables colored output mode
+```
+
+
+
+
+Utility provides the following interface: usage: rss_reader.py source [-h] [--version] [--json] [--verbose] [--limit LIMIT] [--date] [--to-html] [--to-pdf] [--colorize ]
+
+Pure Python command-line RSS reader.
+
+positional arguments: source Input your RSS-link hear. Your link have to start with "https://"
+
+JSON option prints news to stdout. Example:
+
+```
+{
+    "rss_url": "https://people.onliner.by/feed",
+    "channel_title": "Лайфстайл Onlíner",
+    "channel_link": "https://people.onliner.by/",
+    "item_title": "Украина, 126-й день",
+    "item_pubdate": "2022-06-29 08:08:45",
+    "item_description": "Вчера Минобороны России объяснило, что произошло в украинском Кременчуге: с самолетов высокоточным оружием был обстрелян склад с американской техникой. Украинское Минобороны публикует все новые снимки поставок... Продолжаем хронику. Читать далее…",
+    "item_link": "https://people.onliner.by/2022/06/29/ukraina-126-j-den",
+    "item_image": "https://content.onliner.by/news/thumbnail/1fee0b8254c5c24cf728800331bf6f5a.jpeg"
+}
+```
+## Installing the package (if Python is installed, check the version python=3.10)
+
+
+```
+create a folder, put files fo folder
+create a virtual environment 
+download virtualenv for win 'pip install virtualenv' and for Ubuntu 'sudo apt-get install python3-venv'
+create the env for win 'python -m venv venv' and for linux '/usr/bin/python3 -m venv env')
+activate the environment (for win 'env\Scripts\activate.bat', for Ubuntu 'source env/bin/activate')
+pip install -r requirements.txt
+pip setup.py install in folder with files
+The package is ready to use
+```
+
+### Note: if you are using Apple MacBook (or iMac) with M1(ARM) CPU converting to PDF works without pictures. It is bug of ARM architecture

Sholomitski Dmitry/rss_reader/Tests_files/dict_files/dict_1.txt

@@ -0,0 +1,8 @@
+[{'channel_link': 'https://www.yahoo.com/news',
+  'channel_title': 'Yahoo News - Latest News  Headlines',
+  'item_description': 'Description not provided',
+  'item_image': 'https://s.yimg.com/uu/api/res/1.2/Kn3F_gIJwe0a3uIOU.Tb2w--~B/aD0yMzgxO3c9MzU3MTthcHBpZD15dGFjaHlvbg--/https://media.zenfs.com/en/ap.org/4a35cff443aaabc2b49d94a5e7672369',
+  'item_link': 'https://news.yahoo.com/spit-disrespect-arrive-wimbledon-tennis-220151441.html',
+  'item_pubdate': '2022-06-28 22:01:51',
+  'item_title': "Spit, 'disrespect' arrive at Wimbledon as tennis turns ugly",
+  'rss_url': 'https://test_url.com/xml'}]

Sholomitski Dmitry/rss_reader/Tests_files/dict_files/dict_10.txt

@@ -0,0 +1,10 @@
+[{'channel_link': 'https://www.bbc.co.uk/news/',
+  'channel_title': 'BBC News - World',
+  'item_description': "According to US intelligence assessments, Russia's "
+                      'invasion is likely to grind on for some time.',
+  'item_image': 'image is not provided',
+  'item_link': 'https://www.bbc.co.uk/news/world-europe-61990495?at_medium=RSS&at_campaign=KARANGA',
+  'item_pubdate': '2022-06-30 03:00:59',
+  'item_title': 'Russia invasion: Putin still wants to take most of Ukraine - '
+                'US',
+  'rss_url': 'https://test_url.com/xml'}]

Sholomitski Dmitry/rss_reader/Tests_files/dict_files/dict_11.txt

@@ -0,0 +1,10 @@
+[{'channel_link': 'https://vse.sale/',
+  'channel_title': 'Газета ВСЁ',
+  'item_description': 'Празднование Дня города прошло на высоком уровне - была '
+                      'составлена насыщенная программа, соблюдались меры '
+                      'безопасности и контроля, оперативно убран мусор.',
+  'item_image': 'image is not provided',
+  'item_link': 'https://vse.sale/news/view/37519',
+  'item_pubdate': '2022-06-28 19:52:00',
+  'item_title': 'Проблема перегороженных улиц',
+  'rss_url': 'https://test_url.com/xml'}]

Sholomitski Dmitry/rss_reader/Tests_files/dict_files/dict_2.txt

@@ -0,0 +1,18 @@
+[{'channel_link': 'https://news.google.com/?hl=en-US=US=US:en',
+  'channel_title': 'Top stories - Google News',
+  'item_description': 'Liz Cheney Calls Trump ‘a Domestic Threat That We Have '
+                      'Never Faced Before’The New York TimesLiz Cheney says US '
+                      "is 'confronting a domestic threat' in Donald "
+                      'TrumpCNNOpinion | Trump wanted loyalty above all. He '
+                      'got just the opposite.The Washington PostMSNBC’s '
+                      "Nicolle Wallace says Liz Cheney is targeting 'Trumpism' "
+                      'like her father targeted terrorism after 9/11Fox '
+                      'NewsRepublicans must choose Trump or the Constitution, '
+                      "Liz Cheney warns, describing 'threat' like no otherABC "
+                      'News',
+  'item_image': 'image is not provided',
+  'item_link': 'https://news.google.com/__i/rss/rd/articles/CBMiS2h0dHBzOi8vd3d3Lm55dGltZXMuY29tLzIwMjIvMDYvMjkvdXMvcG9saXRpY3MvbGl6LWNoZW5leS1zcGVlY2gtdHJ1bXAuaHRtbNIBAA?oc=5',
+  'item_pubdate': '2022-06-30 03:02:32',
+  'item_title': 'Liz Cheney Calls Trump ‘a Domestic Threat That We Have Never '
+                'Faced Before’ - The New York Times',
+  'rss_url': 'https://test_url.com/xml'}]

Sholomitski Dmitry/rss_reader/Tests_files/dict_files/dict_3.txt

@@ -0,0 +1,11 @@
+[{'channel_link': 'https://www.nytimes.com/section/world',
+  'channel_title': 'NYT > World News',
+  'item_description': 'Applying the lessons of taekwondo, the regional '
+                      'military leader has rallied the embattled southern city '
+                      'of Mykolaiv.',
+  'item_image': 'image is not provided',
+  'item_link': 'https://www.nytimes.com/2022/06/30/world/europe/ukraine-vitaliy-kim-mykolaiv.html',
+  'item_pubdate': '2022-06-30 09:00:24',
+  'item_title': 'Vitaliy Kim, Master Motivator and Symbol of Ukraine’s '
+                'Resistance',
+  'rss_url': 'https://test_url.com/xml'}]