README

WaterCom

********************************************************************************
	0 - Contents
********************************************************************************

1 - COMPONENTS: the files that make up this system

	I.	index.html
	II.	submit.php
	III.	uw.py
	IV.	create.sql
	V.	auto_generate_plots.py

2 - SETUP: instructions for setup on Ubuntu 14.04

3 - TODO: to-do list

	I.	Bugs
	II.	Features

********************************************************************************
	1 - COMPONENTS
********************************************************************************

I. index.html - web interface

	This is a form that takes input values from the user, to be used in an
	experiment:

	1. transmission power: max, min, step
	2. blocks/packet: max, min, step
	3. transmission mode: max, min, step
	4. number of trials (repetitions)
	5. test data: local file to transfer
	6. email address of user

	Transmission power, blocks/packet, and transmission mode will define the
	range of values to test, and the script will transmit the test data
	under every possible set of parameters, for the number of repetitions
	specified. Test data is a user-specified local file that is uploaded to
	our server.

	[!] TODO: implement sending of results via email back to users

	[!] TODO: may want to have user accounts or portals, instead of emailing
	results. A simple way to do this would be to give users a key when they
	submit a job, and have a form on index.html where they can enter this
	key to see results of their experiments.

	[!] TODO: add CSS/Javascript for nicer user interface

II. submit.php - form validation and job enqueueing

	This script validates the form input, sanitizes SQL, and submits a job
	to the input queue on the server.

	[!] TODO:	$ cat -n submit.php | grep TODO

III. uw.py - the main test script

	The test script is a Python file that is adopted from the RUN_CHINFO.SH
	and SUB_PKT.SH shell scripts. On a schedule, it will retrieve recently
	enqueued experiments (rows in InputQueue that were submitted since the
	last batch of experiments was run) from the database, run the test
	script on the fields of each row, and store results in the database
	(Results table).

	During each experiment, a logfile is kept with information about the
	success or failure of each packet transmission; whether or not data was
	mutated or lost in transmission; number of bytes transmitted with each
	packet; current values of trial number, transmission power, mode, and
	blocks/packet.

	[!] TODO:	$ cat -n uw.py | grep TODO


IV. create.sql - initialize the database


	The database (called 'UWNet') will store the parameters that users
	submit with the form, and the results of the tests. There are two
	tables. The InputQueue table will store each form submission. The
	Results table will store the total delay time, number of lost packets,
	and number of retransmissions made during each transmission of the test
	data.

	a. InputQueue

	+----+------+------+------+------+------+------+------+------+------+--/
	| id | mpwr | lpwr | ppwr | mbkn | lbkn | pbkn | mmod | lmod | pmod |  
	+----+------+------+------+------+------+------+------+------+------+--/

	/--+------+----------+-------+---------------+------------+--/
	   | rptt | testData | email | dateSubmitted | exitStatus |
	/--+------+----------+-------+---------------+------------+--/

	/--+---------------+-----------+----------------+
	   | dateCompleted | emailSent | plotsGenerated |
	/--+---------------+-----------+----------------+

	id		primary key

	mpwr/lpwr/ppwr	max, min, step power

	mbkn/lbkn/pbkn  max, min, step blocks per packet

	mmod/lmod/pmod  max, min, step transmission mode

	rptt		number of times to repeat each experiment

			[!] TODO: May want to store the previous ten numerical
			parameters in JSON

	testData	uploaded file (path on our server)

	email		email to send test results to

	dateSubmitted	date and time of form submission

	exitStatus	the exit status of the test script, e.g. if it ran each
			experiment correctly

			[!] TODO: specify exit status codes in uw.py

	dateCompleted	date and time of experiment completion

	emailSent	TRUE if results have been sent to the email specified in
			the queue table, FALSE otherwise 

	plotsGenerated	TRUE if the plots for this experiment have been
			generated, FALSE otherwise

			[!] TODO: may no longer need plotsGenerated if/when
			interactive user portal comes


		b. Results

	+--------------+------------+---------+
	| experimentId | parameters | results |
	+--------------+------------+---------+

	experimentId	foreign key for id in InputQueue, there will be a row
			for each combination of parameters, for each trial

	parameters	JSON string representing the input parameters

			{ "pwr": $TRANSMISSION_POWER,
			  "bkn": $BLOCKS_PER_PACKET,
			  "mod": $TRANSMISSION_MODE }

	results		JSON string representing the experimental results for
			all trials using a particular parameter combination,
			keyed by the trial number.

			{ "0": { "delay": $DELAY_TIME,
			         "loss":  $PACKET_LOSS_COUNT,
			         "retx":  $PACKET_RETX_COUNT },
	                  "1": ...
	                }

			[!] TODO: calculate bit loss rate (derived from block
			loss rate)


V. auto_generate_plots.py - data plotting


	In order for users to see the results of their test, we will generate
	plots of the data and other summaries/analysis, and send the results to
	them via the email provided when they submitted the form.

	Currently, the script checks the database to see which experiments have
	not been plotted already, generates wireframe plots for them, and saves
	images of the plots in the 'plots' folder.

	It is important to note that this only works when the web form is
	submitted as follows:

	        mpwr == lpwr
	        ppwr == { any positive integer }
	        mmod == 5
	        lmod == 1
	        pmod == 1
	        mbkn == 16
	        lbkn == 1
	        pbkn == 1
	
	The next maintainer(s) must fix this plotting script to plot arbitrary
	data sets.


********************************************************************************
	2 - SETUP
********************************************************************************


WaterCom Setup Guide for Ubuntu 14.04 (other OSes may require some tweaks)

1. Ensure that your machine has the following software packages installed:

	Apache/2.4.7
	mysql Ver 14.14 Distrib 5.5.44, 
	PHP 5.5.9-1ubuntu4.11 (cli) 
	Python 2.7.6
	pip 7.1.2 (Python package manager)

...and these Python modules:

	matplotlib
	mpl_toolkits.mplot3d
	mysql.connector
	numpy
	pylab
	scipy
	serial

2. Move the folder 'UWNet-web-interface' inside the public web folder; on
Ubuntu, this is '/var/www/html'. Then:

	$ cd /var/www/html/UWNet-web-interface

3. Create the MySQL database with:

	$ mysql -u [username] -h [localhost] -p[pass] < create.sql

4. To access the form, point a web browser to 'localhost/UWNet-web-interface'.
Enter the values requested and choose a file to upload.

5. With the AquaSeNT modems powered on, connect them to your machine via USB.
To run the submitted experiments:

	$ python uw.py

6. To plot results:

	$ python auto_generate_plots.py

[!] NOTE: CREATING WIREFRAME PLOTS IS ONLY POSSIBLE WHEN THE FORM IS SUBMITTED
WITH VALUES:

	mpwr == lpwr
	ppwr == { any positive integer }
	mmod == 5
	lmod == 1
	pmod == 1
	mbkn == 16
	lbkn == 1
	pbkn == 1
	rptt == 1

THE MAINTAINER OF THIS CODE NEEDS TO FIX auto_generate_plots.py !!!

********************************************************************************
	3 - TODO
********************************************************************************

I. Bugs

In each file I have noted any TODOs that should be done. To find these:

	$ cat -n [FILE] | grep TODO

Otherwise, here are known problems:

	1. The plotting script auto_generate_plots.py only generates correct
	wireframe plots when the form is submitted with certain values; i.e.:

	mpwr == lpwr
	ppwr == { any positive integer }
	mmod == 5
	lmod == 1
	pmod == 1
	mbkn == 16
	lbkn == 1
	pbkn == 1
	rptt == 1

	It is not known if it will succeed with any other values, and needs to
	be thoroughly tested. The maintainer(s) will have to fix the plotting
	script by removing hard-coded values, and figure out how to do wireframe
	plotting the correct way. Generating scatter plots is easier, and the
	code exists in the script already (commented out).

	2. The main test script does not do error reporting very well. There
	should be an error code system that reports to the database if anything
	went wrong during the execution of an experiment (e.g. database
	connection error). Need to decide what is worth reporting. Make sure
	this comes up in the error logs too!

	3. This guide does not address hosting on a live server. It has only
	been hosted locally so far.

II. Features

Here are some ideas for where to go from here:

	1. Decide how to deliver results to user. Can either have user accounts
	which they setup on the main page, or can give them a key upon form
	submission, which they can use to log on and view results for that
	experiment. Or, can simply generate plot files and email to them.

	2. Setup job scheduling for the python scripts. This is simple:

		$ man crontab

	3. Add CSS styling and/or JavaScript to build a nicer or easier-to-use
	front-end.

	4. Improve user experience. Maybe want to do form validation on the
	front-end in additioon to back-end, or fill the form with default values 
	initially. Give better feedback when user enters invalid form data
	(change submit.php).