Skip to content

Dev #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
swatopar wants to merge 32 commits into
base: master
Choose a base branch
from
Open

Dev #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
c1f3d3b
Split classes into seperate files
swatopar Oct 3, 2019
c1000b5
Converted namespaces and project names to Pascal case
swatopar Oct 3, 2019
32489cd
Parser can now parse and calculate total bandwidth in any single log …
heobim10 Nov 5, 2019
1433df0
Merge pull request #7 from social-experiments/feature-add-bandwidth
swatopar Nov 9, 2019
ad6f5ba
Not fully tested, but getting something. Missing modules, though
swatopar Nov 9, 2019
20938d8
Undid changes in parserapp (shouldn't have commit them in the first p…
swatopar Nov 9, 2019
a0ffaef
Hopefully will add all all but total bandwidth
swatopar Nov 9, 2019
0b79a1b
no time
swatopar Nov 9, 2019
dd84b5b
Storing total as a dummy site
swatopar Nov 12, 2019
04df61d
Uncommented uploading the total
swatopar Nov 12, 2019
03d8331
Total now has more identifying information (ie. date, partition key)
swatopar Nov 12, 2019
efbebab
Added device id to total bandwidth id
swatopar Nov 12, 2019
c7ba654
total now has URL as the empty string
swatopar Nov 13, 2019
e0321b8
Total bandwidth url reported as /dev/null
swatopar Nov 13, 2019
70abedc
Changed format for total bandwidth date
swatopar Nov 13, 2019
ba65b5c
Renamed total's url to 'total bandwidth':
swatopar Nov 13, 2019
f068107
Added dev power BI
swatopar Nov 13, 2019
9788010
Added master Power BI
swatopar Nov 13, 2019
3e15af1
Add logic to parse json and tokens lines, still need to check for bug…
heobim10 Dec 4, 2019
bd951f2
Fixed json line parsing error, added necessary packages to be able to…
heobim10 Dec 6, 2019
e6bcbd4
Fixed data substring call
swatopar Dec 9, 2019
664c191
Merge pull request #12 from social-experiments/feature-lines-parsing
swatopar Dec 9, 2019
5aa30c3
Updated test's newtonsoft package
swatopar Dec 9, 2019
d4867ca
Updated parserfn's newtonsoft pacakage
swatopar Dec 9, 2019
3cc19a6
Updating parsertest newtonsoft
swatopar Dec 9, 2019
e21d57d
Put everything on newtonsoft.json 11.02
swatopar Dec 9, 2019
aa4bab0
Merge branch 'dev' of https://github.com/social-experiments/websrvlog…
swatopar Dec 9, 2019
f8bf062
dev PowerBI updated
swatopar Dec 9, 2019
3555795
fixed top n url graph
tfrancisco Jan 15, 2020
6e7ef83
Power BI issues fixed
tfrancisco Jan 22, 2020
97f54ff
small renaming change
tfrancisco Jan 22, 2020
cec4089
some more small changes
tfrancisco Jan 22, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added World Possible Dev.pbix
View file
Open in desktop
Binary file not shown.
Binary file added World Possible Master.pbix
View file
Open in desktop
Binary file not shown.
38 changes: 30 additions & 8 deletions parserapp/access4.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,30 @@
192.168.1.10 - - [24/Jul/2019:18:51:16 +0000] "GET /modules/en-teachertraining/index.html HTTP/1.1" 200 13480 "http://192.168.1.150/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"
192.168.1.10 RACHEL 80 [24/Jul/2019:18:51:16 +0000] "GET /modules/en-teachertraining/teachertraining-logo.jpg HTTP/1.1" 200 13480 "http://192.168.1.150/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"
192.168.1.10 - - [24/Jul/2019:18:52:16 +0000] "GET /modules/en-kolibri-index/kolibri-logo.svg HTTP/1.1" 200 10220 "http://192.168.1.150/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"
192.168.1.10 RACHEL 80 [24/Jul/2019:18:52:16 +0000] "GET /modules/en-kolibri-index/kolibri-logo.svg HTTP/1.1" 200 10220 "http://192.168.1.150/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"
192.168.1.10 - - [24/Jul/2019:18:53:16 +0000] "GET /modules/en-wikipedia/enwiki.png HTTP/1.1" 200 20616 "http://192.168.1.150/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"
192.168.1.10 RACHEL 80 [24/Jul/2019:18:53:16 +0000] "GET /modules/en-wikipedia/enwiki.png HTTP/1.1" 200 20616 "http://192.168.1.150/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"
192.168.1.10 - - [24/Jul/2019:18:53:16 +0000] "GET /modules/en-wikipedia_for_schools/wfs_logo_smooth.jpg HTTP/1.1" 200 31908 "http://192.168.1.150/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"
192.168.1.10 RACHEL 80 [24/Jul/2019:18:53:16 +0000] "GET /modules/en-wikipedia_for_schools/wfs_logo_smooth.jpg HTTP/1.1" 200 31908 "http://192.168.1.150/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET / HTTP/1.1","status": "200","body_bytes_sent":"4821","request_time":"0.394","http_referrer":"-","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET / HTTP/1.1" 200 4821 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /modules/en-asst_medical/Logo-small.png HTTP/1.1","status": "200","body_bytes_sent":"9043","request_time":"0.000","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /modules/en-asst_medical/Logo-small.png HTTP/1.1" 200 9043 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /modules/en-educate/logo.png HTTP/1.1","status": "200","body_bytes_sent":"28560","request_time":"0.000","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /modules/en-educate/logo.png HTTP/1.1" 200 28560 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /modules/en-catdogbooks/logo.png HTTP/1.1","status": "200","body_bytes_sent":"72051","request_time":"0.028","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /modules/en-catdogbooks/logo.png HTTP/1.1" 200 72051 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /js/jquery-1.10.2.min.js HTTP/1.1","status": "200","body_bytes_sent":"93107","request_time":"0.015","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /js/jquery-1.10.2.min.js HTTP/1.1" 200 93107 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /css/normalize-1.1.3.css HTTP/1.1","status": "200","body_bytes_sent":"9559","request_time":"0.000","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /css/normalize-1.1.3.css HTTP/1.1" 200 9559 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /modules/en-edison/Edison-Logo.jpg HTTP/1.1","status": "200","body_bytes_sent":"10914","request_time":"0.000","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /modules/en-edison/Edison-Logo.jpg HTTP/1.1" 200 10914 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /modules/en-afristory/af.png HTTP/1.1","status": "200","body_bytes_sent":"17787","request_time":"0.000","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /modules/en-afristory/af.png HTTP/1.1" 200 17787 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /css/ui-lightness/jquery-ui-1.10.4.custom.min.css HTTP/1.1","status": "200","body_bytes_sent":"17114","request_time":"0.000","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /css/ui-lightness/jquery-ui-1.10.4.custom.min.css HTTP/1.1" 200 17114 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /modules/en-blockly-games/logo.png HTTP/1.1","status": "200","body_bytes_sent":"8954","request_time":"0.000","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /modules/en-blockly-games/logo.png HTTP/1.1" 200 8954 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /js/jquery-ui-1.10.4.custom.min.js HTTP/1.1","status": "200","body_bytes_sent":"60845","request_time":"0.011","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /js/jquery-ui-1.10.4.custom.min.js HTTP/1.1" 200 60845 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /modules/en-bookdash/logo.png HTTP/1.1","status": "200","body_bytes_sent":"12730","request_time":"0.000","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /modules/en-bookdash/logo.png HTTP/1.1" 200 12730 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:25 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /css/style.css HTTP/1.1","status": "200","body_bytes_sent":"4171","request_time":"0.000","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:25 +0000] "GET /css/style.css HTTP/1.1" 200 4171 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:26 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /modules/en-causebooks/osu-logo.png HTTP/1.1","status": "200","body_bytes_sent":"65596","request_time":"0.062","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:26 +0000] "GET /modules/en-causebooks/osu-logo.png HTTP/1.1" 200 65596 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
{"time_local":"23/Sep/2019:22:23:26 +0000","remote_addr":"192.168.88.238","remote_user":"-","request":"GET /modules/en-binofino/B&NFiles/Bino&Fino5.png HTTP/1.1","status": "200","body_bytes_sent":"136122","request_time":"0.062","http_referrer":"http://my.content/","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"}
192.168.88.238 RACHEL 80 [23/Sep/2019:22:23:26 +0000] "GET /modules/en-binofino/B&NFiles/Bino&Fino5.png HTTP/1.1" 200 136122 "http://my.content/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
2 changes: 1 addition & 1 deletion parserapp/parserapp.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\parserlib\parserlib.csproj" />
<ProjectReference Include="..\parserlib\ParserLib.csproj" />
</ItemGroup>

</Project>
13 changes: 13 additions & 0 deletions parserfn/AccessLogSummary.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
using System;
using System.Collections.Generic;
using System.Text;

namespace ParserFn
{
class AccessLogSummary
{
public string DeviceId { get; set; }
public string AccessDate { get; set; }
public long Bandwidth { get; set; }
}
}
17 changes: 17 additions & 0 deletions parserfn/AccessSitesLog.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
using System;
using System.Collections.Generic;
using System.Text;

namespace ParserFn
{
public class AccessedSitesLog
{
public string id { get; set; }
public string PartitionKey { get; set; }
public string Date { get; set; }
public string Url { get; set; }
public long Bandwidth { get; set; }
public long MainModuleCount { get; set; }
public long SubModuleCount { get; set; }
}
}
30 changes: 18 additions & 12 deletions parserfn/JsonToDB.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,10 @@
using Microsoft.Azure.Documents;
using System.Collections.Generic;

namespace parserfn
namespace ParserFn
{
// Cosmos DB Entity
public class AccessedSitesLog
{
public string id { get; set; }
public string PartitionKey { get; set; }
public string Date { get; set; }
public string Url { get; set; }
public long MainModuleCount { get; set; }
public long SubModuleCount { get; set; }
}


public static class JsonToDB
{
Expand All @@ -42,7 +34,7 @@ public static void Run(
result = serializer.Deserialize<AccessDataDetail>(jsonTextReader);
}
}

//For each AccessData Object in the list - create a record in Cosmos DB
foreach (AccessData accessedsite in result.AccessDetails)
{
Expand All @@ -54,13 +46,27 @@ public static void Run(
Url = accessedsite.ModuleName,
Date = accessedsite.UpLoadTime.ToString("MM/dd/yyyy"),
MainModuleCount = accessedsite.MainModuleCount,
SubModuleCount = accessedsite.SubModuleCount
SubModuleCount = accessedsite.SubModuleCount,
Bandwidth = accessedsite.Bandwidth
};

// upload accessedSiteRecord to cosmos db
cosmosOutput.Add(accessedSiteRecord);
log.LogInformation($"Added all to Cosmos DB");
}

// Add general information
var summary = new AccessedSitesLog
{
PartitionKey = result.DeviceId + " " + result.AccessDate,
id = result.DeviceId + " TotalBandwidth",
Url = "Total Bandwidth",
MainModuleCount = -1,
SubModuleCount = -1,
Date = result.AccessDetails[0].UpLoadTime.ToString("MM/dd/yyy"), // should be same date anyways and forces using the correct format. Assumes at least one module was counted
Bandwidth = result.Bandwidth
};
cosmosOutput.Add(summary);
}
}
}
2 changes: 1 addition & 1 deletion parserfn/LogToJson.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
using Microsoft.Extensions.Logging;
using ParserLib;

namespace parserfn
namespace ParserFn
{
public static class LogToJson
{
Expand Down
4 changes: 2 additions & 2 deletions parserfn/parserfn.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp2.2</TargetFramework>
<AzureFunctionsVersion>v2</AzureFunctionsVersion>
Expand All @@ -10,7 +10,7 @@
<PackageReference Include="Newtonsoft.Json" Version="11.0.2" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\parserlib\parserlib.csproj" />
<ProjectReference Include="..\parserlib\ParserLib.csproj" />
</ItemGroup>
<ItemGroup>
<None Update="host.json">
Expand Down
16 changes: 16 additions & 0 deletions parserlib/AccessData.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
using System;
using System.Collections.Generic;
using System.Text;

namespace ParserLib
{
//Represents a record in Access Log
public class AccessData
{
public string ModuleName { get; set; }
public long MainModuleCount { get; set; }
public long SubModuleCount { get; set; }
public DateTime UpLoadTime { get; set; }
public long Bandwidth { get; set; }
}
}
17 changes: 17 additions & 0 deletions parserlib/AccessDataDetail.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
using System;
using System.Collections.Generic;
using System.Text;

namespace ParserLib
{
//AccessDataDetail Object to be serialized into JSON - One per day
public class AccessDataDetail
{
public string DeviceId { get; set; }
public string AccessDate { get; set; }
public DateTime StartTime { get; set; }
public DateTime EndTime { get; set; }
public long Bandwidth { get; set; }
public List<AccessData> AccessDetails { get; set; } = new List<AccessData>();
}
}
32 changes: 32 additions & 0 deletions parserlib/AzureBlobStorage.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.Storage;
using Microsoft.Azure.Storage.Blob;
using Newtonsoft.Json;
using System.Text.RegularExpressions;

namespace ParserLib
{
//Utility Class to Upload to Blob
public class AzureBlobStorage : IStorage
{
public async Task Create(Stream stream, string path, string containerName, string storageConnectionString)
{
CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);

var blobClient = account.CreateCloudBlobClient();

var blobContainer = blobClient.GetContainerReference(containerName);
await blobContainer.CreateIfNotExistsAsync();

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference(path);
await blockBlob.UploadFromStreamAsync(stream);

}
}
}
12 changes: 12 additions & 0 deletions parserlib/DateRange.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
using System;
using System.Collections.Generic;
using System.Text;

namespace ParserLib
{
public class DateRange
{
public DateTime StartTime { get; set; }
public DateTime EndTime { get; set; }
}
}
19 changes: 19 additions & 0 deletions parserlib/IStorage.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.Storage;
using Microsoft.Azure.Storage.Blob;
using Newtonsoft.Json;
using System.Text.RegularExpressions;

namespace ParserLib
{
public interface IStorage
{
Task Create(Stream stream, string path, string containerName, string storageConnectionString);
}
}
22 changes: 22 additions & 0 deletions parserlib/LineDataDetail.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
using System;
using System.Collections.Generic;
using System.Text;

namespace ParserLib
{
//contains information taken out of each line of the log
public class LineDataDetail
{
public string ClientIP { get; set; }
public string Date { get; set; }
public string Module { get; set; }
public long Bandwidth { get; set; }

//note whether this particular line is of JSON type or of "tokenized" type
//these are the only types of log lines that we know of, for now
public string LineType { get; set; }

//to account for some specific condition in "tokenized" lines that the lines must be skipped
public bool Skip { get; set; }
}
}
Loading