From 6d51e9a26227da4c511d055ae77ccd366819ae4e Mon Sep 17 00:00:00 2001 From: Aaron Crosman Date: Tue, 31 Oct 2023 09:56:26 -0500 Subject: [PATCH 1/5] Initial edits --- README.md | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ce6d6dc..362acc5 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,33 @@ Salesforce Admins, Developers, and Consultants need to populate test environment ### Solution We propose a combination of tooling, documentation, and advocacy efforts that offer technical and recommended best practices the problem described above. By developing tools like Snowfakery, as well as documenting existing tools and methods, we empower Admins, Developers, and Consultants to populate Salesforce environments with valid data sets. -### Project Vision/Goals +## Sub-Projects + +The Data Generation Toolkit Project consists of several sub-projects, each focused on one area of the larger challenge we are addressing. + +### Snowfakery Recipes + +Our largest project is our collection of Snowfakery recipes meant to be used for learning and extensions. There are recipes for Salesforce Core, NPSP, EDA, and several open source commons projects. + +### Guides + +We have generated several guides to help support admins and others use need to seed sandboxes or use Snowfakery for other projects. + +* Meet Alex +* Snwofakery Best Practices (Coming soon) +* Snowfakery cheatsheet +* Using Recipes +* Contributing Recipes + +### Education Fakery Provider + +A Python-based Faker project to create higher education fake data. Included in Snowfakery since 3.6. Also avialable as a Python extension that can be used with the main Faker library. + +### Nonprofit Fakery Provider + +A Python-based Faker project to create nonprofit-related fake data. Included in Snowfakery since 3.6. Also avialable as a Python extension that can be used with the main Faker library. + +## General Project Vision/Goals - Every Admin, Developer, and Consultant has access to test environments populated with valid data - Ability to generate data sets that follow a story - Ability to generate data sets that are valid for small, medium, and large data volumes (including LDV) for testing purposes @@ -22,7 +48,6 @@ We propose a combination of tooling, documentation, and advocacy efforts that of - Open source and free tools facilitate not only data generation, but also sharing best practices *about* data generation between orgs - High quality documentation guides users to the appropriate method for their data generation goals and how to execute that method - Support for declarative and developer focused solutions -- Striving for admin-friendly UI for creating data sets [tentatively titled SnowMakery] - This project places a particular emphasis on leadership development, with active participation from many community members who are underrepresented in technology, less familiar with code solutions, and/or early in their career. Contributors who "stick" can be described as: ambitious, stubborn, playful, creative - The Data Generation Toolkit leadership team also maintains the [Snowfakery Recipe Repo](https://github.com/SFDO-Community-Sprints/Snowfakery-Recipe-Templates) From 628234dc88390442a0c123fd517147762928ab94 Mon Sep 17 00:00:00 2001 From: Aaron Crosman Date: Wed, 1 Nov 2023 09:00:56 -0500 Subject: [PATCH 2/5] Readme refactor --- README.md | 67 +++++++++++++++++++------------------------------------ 1 file changed, 23 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index 362acc5..5055563 100644 --- a/README.md +++ b/README.md @@ -27,20 +27,21 @@ Our largest project is our collection of Snowfakery recipes meant to be used for We have generated several guides to help support admins and others use need to seed sandboxes or use Snowfakery for other projects. * Meet Alex -* Snwofakery Best Practices (Coming soon) -* Snowfakery cheatsheet +* Snowfakery Best Practices (Planned) +* Snowfakery cheat sheet (planned) * Using Recipes * Contributing Recipes ### Education Fakery Provider -A Python-based Faker project to create higher education fake data. Included in Snowfakery since 3.6. Also avialable as a Python extension that can be used with the main Faker library. +A Python-based Faker project to create higher education fake data. Included in Snowfakery since 3.6. Also available as a Python extension that can be used with the main Faker library. ### Nonprofit Fakery Provider -A Python-based Faker project to create nonprofit-related fake data. Included in Snowfakery since 3.6. Also avialable as a Python extension that can be used with the main Faker library. +A Python-based Faker project to create nonprofit-related fake data. Included in Snowfakery since 3.6. Also available as a Python extension that can be used with the main Faker library. ## General Project Vision/Goals + - Every Admin, Developer, and Consultant has access to test environments populated with valid data - Ability to generate data sets that follow a story - Ability to generate data sets that are valid for small, medium, and large data volumes (including LDV) for testing purposes @@ -48,47 +49,15 @@ A Python-based Faker project to create nonprofit-related fake data. Included in - Open source and free tools facilitate not only data generation, but also sharing best practices *about* data generation between orgs - High quality documentation guides users to the appropriate method for their data generation goals and how to execute that method - Support for declarative and developer focused solutions -- This project places a particular emphasis on leadership development, with active participation from many community members who are underrepresented in technology, less familiar with code solutions, and/or early in their career. Contributors who "stick" can be described as: ambitious, stubborn, playful, creative -- The Data Generation Toolkit leadership team also maintains the [Snowfakery Recipe Repo](https://github.com/SFDO-Community-Sprints/Snowfakery-Recipe-Templates) - -### Working Shared Definitions -- Test Environments: this project supports working in sandbox, scratch orgs, or dev orgs. Certain use cases for production orgs are valid, but should be attempted with caution and care! -- Data Set: a collection of related records that include multiple objects and fields. -- Valid test data: the resulting data set produced through these methods should meet user-defined critiera, including matching org schema, volume of records, support for record types, custom fields, records related to other records etc. - -#### Example Use Cases We Aim to Support: -- Quality assurance (QA) testing an org populated with permutations of all, or nearly all, the types of data that are relevant to the project. -- Be able to reliably and easily load sample data into any connected Salesforce org -- Have a data set for demos, potentially with the ability to add specific sets of data that could be used for story based training materials. -- Have a data set that ensures the privacy of people represented in the dataset (for example, not real names). -- Have data sets at scales that allow for testing bulk data processing. - -### Ongoing Task Streams: - -Currently the project team has several major efforts for our work: - -- [Maintaining a repository of community-sourced Snowfakery recipes](https://github.com/SFDO-Community-Sprints/Snowfakery-Recipe-Templates) -- Documenting declarative methods and tools to seed Sandboxes with valid data -- Evangelizing Snowfakery through [public events, blog posts,](https://github.com/SFDO-Community-Sprints/DataGenerationToolkit/wiki/Talks---Webinars) - -- On hold: Creation of _Snowmakery_ a community supported tool that will empower easy creation of data sets tailored to an org's needs. - -### Project Accomplishments: - -- [Project Personas](https://raw.githubusercontent.com/SFDO-Community-Sprints/DataGenerationToolkit/master/Assets/DataGenPersonas_202102.pdf): a collection of people we target our documentation about and to. -- [Meet Alex](https://sfdo-community-sprints.github.io/DataGenerationToolkit/DataGenGuide): a guide to populating sandboxes with useful data -- [Snowfakery Example Library](https://github.com/SFDO-Community-Sprints/Snowfakery-Recipe-Templates): a growing collection of Snowfakery data generator recipes for Salesforce. We keep our most recent full notes from meetings and Sprints in the [project wiki](https://github.com/SFDO-Community-Sprints/DataGenerationToolkit/wiki). -#### Proof of Concept Code: - -During the course of this project some ideas have been tested in code, and to ensure that code isn't lost and doesn't confuse other efforts in this repo they are kept in the proofs-of-concept directory. -Currently those include: +### Working Shared Definitions -1. [A CumulusCI Task](proofs-of-concept/OriginalCciTask) created during this project's first sprint to generate permutations of data. -1. [A web-based Snowfakery recipe editor](proofs-of-concept/SnowmakeryEditor) created during the fall 2020 virtual sprint. This was the second tool named Snowmakery and largely confirmed that this project's UI will bare that name. +- *Test Environments*: this project supports working in sandbox, scratch orgs, or dev orgs. Certain use cases for production orgs are valid, but should be attempted with caution and care! +- *Data Set*: a collection of related records that include multiple objects and fields. +- *Valid test data*: the resulting data set produced through these methods should meet user-defined critiera, including matching org schema, volume of records, support for record types, custom fields, records related to other records etc. ### Project Meetings @@ -106,15 +75,25 @@ To help people identify with specific audiences while creating documentation we Help us give you the thanks you deserve and ensure future contributors know who to contact if they have questions! Please ensure that all contributing members of the team are included. -- Team Leader(s): - - Aaron Crosman (Attain) +Team Leader(s): + - Aaron Crosman (Coastal Cloud) + - Eilleen Kapp + - Allison Letts (Attain Partners) + - Jung Mun - Samantha Shain (William Penn Foundation) - Cassie Supilowski (OneGoal) -- Salesforce Liaison: - - Paul Prescod (Salesforce.org) We also keep a [complete list of contributors](https://github.com/SFDO-Community-Sprints/DataGenerationToolkit/wiki/Project-Contributors). Please add yourself to the list! +### Proof of Concept Code: + +During the course of this project some ideas have been tested in code, and to ensure that code isn't lost and doesn't confuse other efforts in this repo they are kept in the proofs-of-concept directory. + +Currently those include: + +1. [A CumulusCI Task](proofs-of-concept/OriginalCciTask) created during this project's first sprint to generate permutations of data. +1. [A web-based Snowfakery recipe editor](proofs-of-concept/SnowmakeryEditor) created during the fall 2020 virtual sprint. This was the second tool named Snowmakery and largely confirmed that this project's UI will bare that name. + ### Past Project Accomplishments - *Between Feb '21 Virtual Sprint and present* From 16abfae230362a7d281423865c11498fda03a1f4 Mon Sep 17 00:00:00 2001 From: Aaron Crosman Date: Mon, 5 Feb 2024 18:02:30 -0500 Subject: [PATCH 3/5] Adding links to other parts of the project --- README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 5055563..c567311 100644 --- a/README.md +++ b/README.md @@ -26,19 +26,21 @@ Our largest project is our collection of Snowfakery recipes meant to be used for We have generated several guides to help support admins and others use need to seed sandboxes or use Snowfakery for other projects. -* Meet Alex +You can see all the guides on our [guide index](https://sfdo-community-sprints.github.io/DataGenerationToolkit/). + +* [Meet Alex](https://sfdo-community-sprints.github.io/DataGenerationToolkit/DataGenGuide.html) * Snowfakery Best Practices (Planned) * Snowfakery cheat sheet (planned) -* Using Recipes -* Contributing Recipes +* Using Recipes (planned) +* [Contributing Recipes](https://github.com/SFDO-Community-Sprints/Snowfakery-Recipe-Templates/tree/main) ### Education Fakery Provider -A Python-based Faker project to create higher education fake data. Included in Snowfakery since 3.6. Also available as a Python extension that can be used with the main Faker library. +A [Python-based Faker project](https://pypi.org/project/faker-edu/) to create higher education fake data. Included in Snowfakery since 3.6. Also available as a Python extension that can be used with the main Faker library. [Code for this library is found on Github.](https://github.com/SFDO-Community-Sprints/Snowfakery-Edu) ### Nonprofit Fakery Provider -A Python-based Faker project to create nonprofit-related fake data. Included in Snowfakery since 3.6. Also available as a Python extension that can be used with the main Faker library. +A [Python-based Faker project to create nonprofit-related fake data](https://pypi.org/project/faker-nonprofit/). Included in Snowfakery since 3.6. Also available as a Python extension that can be used with the main Faker library. [Code for this library is found on Github.](https://github.com/SFDO-Community-Sprints/Snowfakery-nonprofit) ## General Project Vision/Goals From a80f523f115332a8c8a6bd69499a3feb2ed6a33a Mon Sep 17 00:00:00 2001 From: Aaron Crosman Date: Mon, 5 Feb 2024 21:05:27 -0500 Subject: [PATCH 4/5] Moved older material to own page --- README.md | 38 +------------------------------------- docs/ProjectHistory.md | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 37 deletions(-) create mode 100644 docs/ProjectHistory.md diff --git a/README.md b/README.md index c567311..528e6b3 100644 --- a/README.md +++ b/README.md @@ -87,43 +87,6 @@ Team Leader(s): We also keep a [complete list of contributors](https://github.com/SFDO-Community-Sprints/DataGenerationToolkit/wiki/Project-Contributors). Please add yourself to the list! -### Proof of Concept Code: - -During the course of this project some ideas have been tested in code, and to ensure that code isn't lost and doesn't confuse other efforts in this repo they are kept in the proofs-of-concept directory. - -Currently those include: - -1. [A CumulusCI Task](proofs-of-concept/OriginalCciTask) created during this project's first sprint to generate permutations of data. -1. [A web-based Snowfakery recipe editor](proofs-of-concept/SnowmakeryEditor) created during the fall 2020 virtual sprint. This was the second tool named Snowmakery and largely confirmed that this project's UI will bare that name. - -### Past Project Accomplishments - -- *Between Feb '21 Virtual Sprint and present* - - Moved Snowfakery recipe cookbook into an independent, community maintained repo - - Designed Meeting in a Box for User Group and Conference presentations on sandbox seeding - - Defined use case for seeding a sandbox with a Flow - - Scheduled quarterly, public Snowfakery trainings -- *Between Sept '20 Virtual Sprint and Feb '21 Virtual Sprint* - - Completed first full draft of Data Generation Guide document; including technical edit - - Kicked the tires on Snowfakery, for real - - Made several public presentations - - Implemented Project Boards to track issues and discussion topics - - Survived despite pandemic, fascism, etc. -- *9/23/20-9/24/20 Virtual Sprint* - - Drafted architecture diagrams for a UI to sit on top of Snowfakery. This will be called _Snowmakery!_ There are two proof of concepts that have been initiated. One is inside of the org, the other is in Heroku. - - Analyzed survey data from 75 community members; updated relevant Personas based on survey insights - - Established document outline and Admin Story for documentation project that describes how to Generate Data and Move Data between orgs - - Documented limitations and considerations for Partial Data sampling algorithm and manual steps, third party apps, and code for creating mock data records - - Documented steps for an Admin-audience to use CCI to move data records from one Dev Sandbox to another Dev Sandbox (or any two persistent orgs) - - QA-ed documentation steps for CCI steps (referenced above) -- *3/31/20 -> 4/1/20 virtual sprint* - - Determined that Snowfakery accomplishes many of the original requirements brainstormed at the Philly Sprint (fall 2019) (namely: ability to generate mock data with related tables, random names and values, standard and custom objects, datasets of any size/scale, ability to populate Salesforce orgs) - - Socialized Snowfakery to community members - - Onboarded project leadership from multiple orgs and began application for inclusion in Open Source Commons program - - Refined use cases and differentiated Snowfakery from (1) existing tools in the market (2) proprietary tools at Salesforce (3) Full Sandbox product - - Brainstormed 2+ potential directions for extending Snowfakery to include an admin-friendly web interface - - Reviewed documentation and install steps - - Overhauled ReadMe file ## Additional Useful References @@ -143,3 +106,4 @@ Currently those include: - [NPSP Data Dictionary](https://attain-projects.quip.com/yD1wAsdz1m1Q/NPSP-Public-Data-Dictionary) - [Wave Data Generator](https://github.com/ttse-sfdc/sfdc-wave-data-generator) (generates data for Salesforce org, and builds linkages between objects) - [JSON/YAML Editor](https://json-editor.github.io/json-editor/) +- [Project History](docs/ProjectHistory.md) diff --git a/docs/ProjectHistory.md b/docs/ProjectHistory.md new file mode 100644 index 0000000..fce50c0 --- /dev/null +++ b/docs/ProjectHistory.md @@ -0,0 +1,37 @@ +### Proof of Concept Code: + +During the course of this project some ideas have been tested in code, and to ensure that code isn't lost and doesn't confuse other efforts in this repo they are kept in the proofs-of-concept directory. + +Currently those include: + +1. [A CumulusCI Task](proofs-of-concept/OriginalCciTask) created during this project's first sprint to generate permutations of data. +1. [A web-based Snowfakery recipe editor](proofs-of-concept/SnowmakeryEditor) created during the fall 2020 virtual sprint. This was the second tool named Snowmakery and largely confirmed that this project's UI will bare that name. + +### Past Project Accomplishments + +- *Between Feb '21 Virtual Sprint and present* + - Moved Snowfakery recipe cookbook into an independent, community maintained repo + - Designed Meeting in a Box for User Group and Conference presentations on sandbox seeding + - Defined use case for seeding a sandbox with a Flow + - Scheduled quarterly, public Snowfakery trainings +- *Between Sept '20 Virtual Sprint and Feb '21 Virtual Sprint* + - Completed first full draft of Data Generation Guide document; including technical edit + - Kicked the tires on Snowfakery, for real + - Made several public presentations + - Implemented Project Boards to track issues and discussion topics + - Survived despite pandemic, fascism, etc. +- *9/23/20-9/24/20 Virtual Sprint* + - Drafted architecture diagrams for a UI to sit on top of Snowfakery. This will be called _Snowmakery!_ There are two proof of concepts that have been initiated. One is inside of the org, the other is in Heroku. + - Analyzed survey data from 75 community members; updated relevant Personas based on survey insights + - Established document outline and Admin Story for documentation project that describes how to Generate Data and Move Data between orgs + - Documented limitations and considerations for Partial Data sampling algorithm and manual steps, third party apps, and code for creating mock data records + - Documented steps for an Admin-audience to use CCI to move data records from one Dev Sandbox to another Dev Sandbox (or any two persistent orgs) + - QA-ed documentation steps for CCI steps (referenced above) +- *3/31/20 -> 4/1/20 virtual sprint* + - Determined that Snowfakery accomplishes many of the original requirements brainstormed at the Philly Sprint (fall 2019) (namely: ability to generate mock data with related tables, random names and values, standard and custom objects, datasets of any size/scale, ability to populate Salesforce orgs) + - Socialized Snowfakery to community members + - Onboarded project leadership from multiple orgs and began application for inclusion in Open Source Commons program + - Refined use cases and differentiated Snowfakery from (1) existing tools in the market (2) proprietary tools at Salesforce (3) Full Sandbox product + - Brainstormed 2+ potential directions for extending Snowfakery to include an admin-friendly web interface + - Reviewed documentation and install steps + - Overhauled ReadMe file From 41070f14f147ae20a4a070c696e5d12634f7d511 Mon Sep 17 00:00:00 2001 From: Aaron Crosman Date: Mon, 5 Feb 2024 21:16:57 -0500 Subject: [PATCH 5/5] Heading Update --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 528e6b3..718b8c5 100644 --- a/README.md +++ b/README.md @@ -87,8 +87,7 @@ Team Leader(s): We also keep a [complete list of contributors](https://github.com/SFDO-Community-Sprints/DataGenerationToolkit/wiki/Project-Contributors). Please add yourself to the list! - -## Additional Useful References +## Project References - [Project Meeting Notes](https://github.com/SFDO-Community-Sprints/DataGenerationToolkit/wiki). - [Snowfakery](https://github.com/SFDO-Tooling/Snowfakery)