csv-xmlConduct full UX architecture review

.env.example

@@ -0,0 +1,11 @@
+# Database
+DATABASE_URL=postgresql://user:password@db:5432/csvtoxml
+POSTGRES_USER=user
+POSTGRES_PASSWORD=password
+POSTGRES_DB=csvtoxml
+
+# Redis
+REDIS_URL=redis://redis:6379
+
+# Authentication
+NEXTAUTH_SECRET=generate-a-random-secret-here

.github/workflows/ci.yml

@@ -0,0 +1,44 @@
+name: CI
+
+on:
+  push:
+    branches: ["**"]
+  pull_request:
+    branches: [master]
+
+jobs:
+  python-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+
+      - name: Run tests
+        run: python -m pytest tests/ -v
+
+  web-lint:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: apps/web
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Lint
+        run: npm run lint
+
+      - name: Build
+        run: npm run build

.gitignore

@@ -1,8 +1,27 @@
-venv/ 
-__pycache__/ 
-*.pyc 
-logs/ 
-reports/ 
-*.log 
-*.csv 
-*.xml 
+venv/
+__pycache__/
+*.pyc
+logs/
+reports/
+*.log
+*.csv
+# Keep sample CSVs that the web app serves to users
+!apps/web/public/samples/*.csv
+
+# Keep XSD files in schemas/ but ignore sample XMLs
+*.xml
+!schemas/*.xsd
+
+# Environment
+.env
+
+# Next.js
+apps/web/.next/
+apps/web/node_modules/
+apps/web/.env.local
+*.tsbuildinfo
+
+# Data
+/data/
+uploads/
+instance/

README.md

@@ -1,8 +1,73 @@
 # SBA Counseling and Training Data Conversion Tool
 
-This utility is designed to process, clean, validate, and convert SBA (Small Business Administration) counseling and training data from CSV format into compliant XML files. It ensures the final XML adheres to the strict sequence and data requirements of the SBA NEXUS schemas for **Form 641 (Counseling)** and **Management Training Reports**.
+Converts SBA counseling and training CSV data into XSD-compliant XML files.
 
-The tool includes robust data cleaning, validation reporting, and a fixer utility for pre-existing XML files.
+The tool ships in two forms:
+
+- A **web application** (`apps/web`, `apps/worker`) — recommended for
+  most users. Handles authentication, uploads, preview/mapping,
+  validation reports, job history, and downloads via a browser.
+- A **Python CLI** (`run.py`, `src/`) — the original interactive
+  launcher, useful for power users and for scripting.
+
+-----
+
+## Web App (recommended)
+
+The web app is a Next.js frontend backed by a FastAPI worker, Postgres,
+and Redis — all wired up in `docker-compose.yml`.
+
+### Run it locally
+
+```bash
+cp .env.example .env
+# Edit .env to set DATABASE_URL, NEXTAUTH_SECRET, etc.
+docker compose up
+```
+
+Then open <http://localhost:3000>, create an account, and upload a CSV.
+
+### Download sample CSVs
+
+Sample CSVs for each converter type live under
+`apps/web/public/samples/` and are also linked from the landing page
+and the dashboard empty state inside the app:
+
+- `counseling-sample.csv` — individual counseling sessions (Form 641)
+- `training-sample.csv` — aggregated training events (Form 888)
+- `training-client-sample.csv` — per-attendee rows (Form 641)
+
+### UX documentation
+
+- [`UX_REVIEW.md`](./UX_REVIEW.md) — severity-ranked audit of the
+  web app's user-facing surfaces.
+- [`UX_IMPLEMENTATION_PLAN.md`](./UX_IMPLEMENTATION_PLAN.md) — the
+  phased roadmap that sequences the UX review findings into
+  executable slices.
+- [`TECHNICAL_DEBT.md`](./TECHNICAL_DEBT.md) — code/security debt
+  register, separate from UX concerns.
+
+-----
+
+## Python CLI
+
+-----
+
+## Quick Start (3 steps)
+
+1. **Download** — On the GitHub page, click the green **Code** button → **Download ZIP**. Unzip the folder anywhere on your computer.
+
+2. **Setup** (one time only) — Requires [Python](https://www.python.org/downloads/) (check **"Add Python to PATH"** during install).
+   - **Windows:** Double-click `setup.bat`
+   - **Mac/Linux:** Open a terminal in the folder and run: `pip install -r requirements.txt`
+
+3. **Run** — Put your CSV file in the folder, then:
+   - **Windows:** Double-click `run.bat`
+   - **Mac/Linux:** Open a terminal in the folder and run: `python run.py`
+
+   The tool will walk you through selecting your CSV file, conversion type, and optional XSD validation — no typing commands needed.
+
+Your output XML and validation reports will be saved in the `output/` and `reports/` folders.
 
 -----
 
@@ -24,14 +89,17 @@ The tool includes robust data cleaning, validation reporting, and a fixer utilit
   * **Validation & Reporting**:
       * During conversion, it generates comprehensive validation reports in both CSV and HTML formats, detailing any issues found in the source data.
   * **XML Fixer Utility**:
-      * Includes a standalone script (`fix-sba-xml.py`) to correct element ordering issues in existing XML files that do not conform to the schema.
+      * Includes a standalone script (`fix_sba_xml.py`) to correct element ordering issues in existing XML files that do not conform to the schema.
 
 -----
 
 ## Project Structure
 
 ```
 .
+├── run.py                         # Interactive launcher (start here!)
+├── run.bat                        # Windows double-click shortcut
+├── setup.bat                      # Windows one-time setup
 ├── src/
 │   ├── converters/
 │   │   ├── base_converter.py       # Base class for all converters
@@ -44,8 +112,8 @@ The tool includes robust data cleaning, validation reporting, and a fixer utilit
 │   ├── config.py                   # Central configuration for field mappings, defaults, and validation rules
 │   ├── validation_report.py        # Module for tracking and reporting validation issues
 │   ├── logging_util.py             # Configures application-wide logging
-│   ├── fix-sba-xml.py              # Utility to fix element order in existing SBA XML files
-│   └── xml-validator.py            # Utility to validate XML files against an XSD
+│   ├── fix_sba_xml.py              # Utility to fix element order in existing SBA XML files
+│   └── xml_validator.py            # Utility to validate XML files against an XSD
 ├── tests/
 │   ├── test_counseling_converter.py
 │   ├── test_data_cleaning.py
@@ -85,10 +153,10 @@ The primary entry point for the conversion is `src/main.py`.
 
 ### Fixing an Existing XML File
 
-If you have an XML file that fails validation due to incorrect element order, use the `fix-sba-xml.py` script:
+If you have an XML file that fails validation due to incorrect element order, use the `fix_sba_xml.py` script:
 
 ```bash
-python -m src.fix-sba-xml --file /path/to/your/invalid.xml --output /path/to/output/fixed.xml
+python -m src.fix_sba_xml --file /path/to/your/invalid.xml --output /path/to/output/fixed.xml
 ```
 
 This will re-order the elements to match the schema requirements.

Sample641CouselingRecord-2-14.xml

@@ -0,0 +1,180 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<CounselingInformation>
+
+
+	<CounselingRecord>
+	<Operation></Operation>
+		<PartnerClientNumber>234347</PartnerClientNumber>
+		<Location>
+			<LocationCode>786</LocationCode>
+		</Location> 
+		<ClientRequest>
+			<ClientNamePart1>
+				<Last>Hanks</Last>
+				<First>Jerry</First>
+				<Middle></Middle>
+			</ClientNamePart1>
+			<Email>jomh@gmail.com</Email>
+			<PhonePart1><Primary>2365894123</Primary><Secondary></Secondary></PhonePart1>
+			<AddressPart1>
+				<Street1></Street1>
+				<Street2></Street2> 
+				<City>Austin</City> 
+				<State>Alabama</State>
+<ZipCode>33189</ZipCode>
+	<Zip4Code>2344</Zip4Code>
+				<Country><Code>United States</Code></Country>
+				<PostalCode></PostalCode>
+				<StateOrProvince></StateOrProvince>
+			</AddressPart1>
+
+			<SurveyAgreement>Yes</SurveyAgreement>
+
+			<ClientSignature>
+				<Date>1987-12-12</Date>
+				<OnFile>Yes</OnFile>
+			</ClientSignature>
+		</ClientRequest>
+		<ClientIntake>
+			<Race><Code>White</Code><Code>Asian</Code><SelfDescribedRace></SelfDescribedRace></Race>
+			<Ethnicity>Non Hispanic or Latino</Ethnicity>
+<Sex>Female</Sex>
+			<Disability>No</Disability>
+			<MilitaryStatus>Active Duty</MilitaryStatus>
+			<BranchOfService>Prefer not to say</BranchOfService>
+			<Media>
+				<Code>Magazine/Newspaper</Code>
+
+<Code>Other</Code>
+				<Other>testmedia</Other>
+			</Media>
+			<Internet></Internet>
+			<CurrentlyInBusiness>Yes</CurrentlyInBusiness>
+			<CurrentlyExporting>Yes</CurrentlyExporting>
+			<CompanyName>ABC company</CompanyName>
+			<BusinessType>Real Estate and Rental and Leasing</BusinessType>
+			<BusinessOwnership>
+				<Female>100</Female>
+			</BusinessOwnership>
+			<ConductingBusinessOnline>No</ConductingBusinessOnline>
+
+			<ClientIntake_Certified8a>No</ClientIntake_Certified8a>
+			<Employee_Owned>Yes</Employee_Owned>
+			<TotalNumberOfEmployees>14</TotalNumberOfEmployees>
+			<NumberOfEmployeesInExportingBusiness>0</NumberOfEmployeesInExportingBusiness>
+			<ClientAnnualIncomePart2>
+				<GrossRevenues>0.00</GrossRevenues>
+				<ProfitLoss>0.00</ProfitLoss>
+				<ExportGrossRevenuesOrSales>5660.00</ExportGrossRevenuesOrSales>
+			</ClientAnnualIncomePart2>
+			<LegalEntity>
+				<Code>Other</Code>
+				<Other>Other Counseling</Other>
+			</LegalEntity>
+			<Rural_vs_Urban>Urban</Rural_vs_Urban>
+			<FIPS_Code>54346</FIPS_Code>
+
+			<CounselingSeeking>
+				<Code>Business Plan</Code>
+				<Other></Other>
+			</CounselingSeeking>
+			<ExportCountries><Code>Belgium</Code><Other></Other></ExportCountries>
+
+		</ClientIntake>
+		<CounselorRecord>
+			<PartnerSessionNumber>371786_T50267</PartnerSessionNumber>
+
+			<FundingSource>Resiliency and Recovery Demonstration Grant – CARESRRD</FundingSource>
+
+			<ClientNamePart3>
+				<Last>hanks</Last>
+				<First>tom</First>
+				<Middle></Middle>
+			</ClientNamePart3>
+			<Email>tomh@gmail.com</Email>
+			<PhonePart3><Primary>2365894123</Primary><Secondary></Secondary></PhonePart3>
+			<AddressPart3>
+			<Street1></Street1>
+			<Street2></Street2>
+				<City>Alpharetta</City>
+				<State>Alabama</State>
+				<ZipCode>95928</ZipCode>
+				<Zip4Code>2344</Zip4Code>
+				<Country><Code>United States</Code></Country>
+				<PostalCode></PostalCode>
+				<StateOrProvince></StateOrProvince>
+			</AddressPart3>
+			<VerifiedToBeInBusiness>Yes</VerifiedToBeInBusiness>
+			<ReportableImpact>No</ReportableImpact>
+			<DateOfReportableImpact></DateOfReportableImpact>
+			<CurrentlyExporting>No</CurrentlyExporting>
+			<BusinessStartDatePart3>2021-12-31</BusinessStartDatePart3>
+			<TotalNumberOfEmployees>18</TotalNumberOfEmployees>
+			<NumberOfEmployeesInExportingBusiness></NumberOfEmployeesInExportingBusiness>
+			<ClientAnnualIncomePart3>
+				<GrossRevenues>0.00</GrossRevenues>
+				<ProfitLoss>0.00</ProfitLoss>
+				<ExportGrossRevenuesOrSales>788.00</ExportGrossRevenuesOrSales>
+				<GrowthIndicator>Yes</GrowthIndicator>
+			</ClientAnnualIncomePart3>
+			<ResourcePartnerServiceContributed>
+			<SBALoanAmount>10</SBALoanAmount>
+			<NonSBALoanAmount>10
+			</NonSBALoanAmount>
+			<EquityCapitalReceived>10
+			</EquityCapitalReceived>
+			<SBALoanAmountTxnNmb>80
+			</SBALoanAmountTxnNmb>
+			<NonSBALoanAmountTxnNmb>80
+			</NonSBALoanAmountTxnNmb>
+			<EquityCapitalReceivedTxnNmb>80
+			</EquityCapitalReceivedTxnNmb>
+			<NumberOfContractsReceived>80
+			</NumberOfContractsReceived>
+
+			</ResourcePartnerServiceContributed>
+			<Certifications>
+				<Code>Service-Disabled Veteran-Owned Small Business</Code>
+				<Other></Other>
+			</Certifications>
+			<SBAFinancialAssistance>
+				<Code>Community Advantage</Code>
+				<Code>Other(SBIR, SBIC, 7(a) 504, etc)</Code>
+				<Other>Other SBA Disaster Loan for COVID-19</Other>
+			</SBAFinancialAssistance>
+			<CounselingProvided>
+				<Code>Tax Planning</Code>
+				<Other></Other>
+			</CounselingProvided>
+			<ReferredClient>
+				<Code>SBA Office of International Trade (OIT)</Code>
+				<Other></Other>
+			</ReferredClient>
+			<SessionType>Online</SessionType>
+			<Language>
+				<Code>English</Code>
+				<Other></Other>
+			</Language>
+			<DateCounseled>2024-01-01</DateCounseled>
+			<CounselorName>Paul Bozzo (FC)</CounselorName>
+			<CounselingHours>
+				<Contact>7</Contact>
+				<Prepare>6</Prepare>
+				<Travel>6</Travel>
+			</CounselingHours>
+			<CounselorNotes>Test</CounselorNotes>
+			<ExportCountries>
+			<Code>United States</Code>
+
+			<Other></Other>
+			</ExportCountries>
+			<TrainingSession>
+				<DateTrainingStarted>2024-09-22</DateTrainingStarted>
+				<PartnerTrainingNumber>PartnerTrainingNum</PartnerTrainingNumber>
+				<EmployeesTrained>2</EmployeesTrained>
+				<HoursTrained>3</HoursTrained>
+			</TrainingSession>
+		</CounselorRecord>
+	</CounselingRecord>
+
+</CounselingInformation>