Devops toy sba by pmitros · Pull Request #176 · ArgLab/writing_observer

pmitros · 2024-09-16T12:45:35Z

No description provided.

pmitros · 2024-09-16T12:46:57Z

devops/tasks/config/nginx_toy_sba

@@ -0,0 +1,23 @@
+server {


I don't like hard-coding this. This likely should be another machine config in config/[machine] here or in learning-observer-flock.

pmitros · 2024-09-16T12:47:35Z

devops/tasks/config/sync.csv

 passwd.lo,root:root,644,/home/ubuntu/writing_observer/learning_observer/passwd.lo,"(Generally blank) passwords file"
-lo.sh,ubuntu:ubuntu,744,/home/ubuntu/writing_observer/learning_observer/lo.sh,"Script to start Learning Observer with a nice process name"
-systemd,root:root,644,/etc/systemd/system/learning_observer.service,"Systemd init script"
+toy-sba.sh,ubuntu:ubuntu,744,/home/ubuntu/toy-sba/toy-sba.sh,"Script to start toy-sba"


This will break the system if we're not installing toy_sba, which is usually the case. See how this was handled for dynamic assessment

pmitros · 2024-09-16T12:48:16Z

devops/tasks/orchlib/config.py

+        "git_username": "Your git username",
+        "git_pac": "Your git personal access token",
+        "openai_url": "currently https://canadaeast.api.cognitive.microsoft.com",
+        "openai_deployment_id": "currently PBrostGPT4",


You probably don't want this kind of information in a public git repo.

pmitros · 2024-09-16T12:49:24Z

devops/tasks/orchlib/ubuntu.py


+def install_git_repos(ip):
+
+    git_username = orchlib.config.creds.get('git_username')


Should work if this is not set.

pmitros · 2024-09-16T12:49:47Z

devops/tasks/orchlib/ubuntu.py

+
+    git_username = orchlib.config.creds.get('git_username')
+    git_pac = orchlib.config.creds.get('git_pac')
+    openai_url = orchlib.config.creds.get('openai_url')


We don't require OpenAI to work. Most LO installs won't need it.

pmitros · 2024-09-16T12:50:45Z

devops/tasks/orchlib/ubuntu.py

+    git_pac = orchlib.config.creds.get('git_pac')
+    openai_url = orchlib.config.creds.get('openai_url')
+    openai_deployment_id = orchlib.config.creds.get('openai_deployment_id')
+    openai_api_key = orchlib.config.creds.get('openai_api_key')


All of this should be generic. We shouldn't hardcode to OpenAI or github.

pmitros · 2024-09-16T12:51:37Z

devops/tasks/orchlib/ubuntu.py

+        content = file.read()
+
+    # Replace placeholders with actual credentials
+    content = content.replace('{git_username}', git_username)


Once things are made generic, this is a content.format (or we should check if we're using a templating language like mustache; look at the other code). This kind of code is very bad and brittle.

pmitros · 2024-09-16T12:52:40Z

devops/tasks/scripts/baseline_packages.fab

-git clone https://github.com/ETS-Next-Gen/writing_observer.git
-cd writing_observer/devops/ansible ; sudo ansible-playbook local.yaml
+
+sudo apt update


We should not be installing anything with apt except for ansible and what's needed to run it. The machine config is managed with ansible.

pmitros · 2024-09-16T12:53:02Z

devops/tasks/scripts/baseline_packages.fab

+sudo apt-get -y install git ansible python3-pip python3-venv pipx
+
+# Ensure pipx binaries are available in the PATH
+python3 -m pipx ensurepath


Ditto for pip packages.

pmitros · 2024-09-16T12:54:44Z

devops/tasks/scripts/git_repos.fab

+cd
+
+# Set system variables
+export OPENAI_URL="{openai_url}"


Virtually everything in this file should be config somewhere, as opposed to hardcoding this to be specific to toy_sba. The only package we want a hard-coded install of is learning_observer.

pmitros · 2024-09-16T12:55:39Z

devops/tasks/scripts/python_venv.fab

 cd
 echo . /usr/share/virtualenvwrapper/virtualenvwrapper.sh >> ~/.profile
-source ~/.profile; mkvirtualenv learning_observer
+source ~/.profile; 


We do this two lines later.

pmitros · 2024-09-16T12:56:09Z

devops/tasks/scripts/python_venv.fab

+export WORKON_HOME=$HOME/.virtualenvs
+source /usr/share/virtualenvwrapper/virtualenvwrapper.sh
+
+source ~/.bashrc


This was already done when the shell started. I'm also not sure this persists.

pmitros · 2024-09-16T12:58:07Z

devops/tasks/scripts/python_venv.fab

+source /usr/share/virtualenvwrapper/virtualenvwrapper.sh
+
+source ~/.bashrc
+mkvirtualenv --version


pmitros · 2024-09-16T18:35:31Z

devops/requirements.txt

 pyyaml
 fabric
-
+unzip


Where is this used?

pmitros · 2024-09-16T18:36:10Z

devops/tasks/README.md

+sudo inv initialize [machine]
+inv baseline [ip address]
+inv gitrepos [ip address]
+inv venv [ip address]


Why do these take the ip address instead of the machine?

pmitros · 2024-09-16T18:36:27Z

devops/tasks/config/authorized_keys

+Copy the following into: ~/.ssh/authorized_keys
+
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCJ/ZWcQBJTXHHlmztnwY39sueZIR8CA5lKwBH9f7ra939sS/Gdgn1+OF/YUgFLC00GBtesuxMJB52xp2RaIxkrD56G1j64f3P1He22F8jdRUZ8+12PdhqtPsMD6eWofmyo5l2nNEwvQkwTQqQDRWQxuYBWmflHUKqyxPROSCCJXcq2gjVgvFcQZd8ZD24EKQAU3/T6lFwxeiOBNhD5EL6JQ/1Wc2kax9FNW+H8mVVPE+xhFQoGS/5phP2EoEA4EQ6Lpxc2AT68IsKPueKTK5vlpKl2qwt4u9UVT2aMeWapo/qXi1mJt8puBODkU28ygCK0LBQGr/SSbS+YRw16JXH5
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHmSeXMa7i6PxwTM/U/AHhVwSkWxni78hmxGs+MyZ28Ck6ndHRF5ex2JY/WZMHgJ5Z1ndOZ6riXmdex5gjjiKLpAYgyTR2Dl0aKo52+xIAK7OY6zs9WyV7XgOLDBzvoMewWFp3/2P88oVh2JEInzLe8hkl2npvW9/37ZL3+J9KAg9nQkFn6WPYH7hdJLqnmv25fabmZNsQmekXuz5jPuQbZ4n0zyLkbMxgmYPmgOlsHDO0Bomv9+gpffgWShkDwrXbasEg51OPZjbx/cssr383e0/dHGFPl0B+WRD7Pw796ehyFrE4HHC4nZYI2CS1iurfbPG8CtSUQM9xsAX7uZLl


Please include what the keys are.

…ditional tasks to do some of env fab script

pbrost added 2 commits September 12, 2024 10:54

not all scripts run remotely as expected

4846dde

update to readme

71de24d

pmitros commented Sep 16, 2024

View reviewed changes

ansible change, pushing to toy-sba server for testing

138a407

pmitros commented Sep 16, 2024

View reviewed changes

devops/requirements.txt

pyyaml

fabric

unzip

Copy link

Contributor Author

pmitros Sep 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this used?

pmitros commented Sep 16, 2024

View reviewed changes

pbrost and others added 8 commits September 23, 2024 20:41

moved code to ansible yaml files

3ee2118

pushing ansible changes

9b79ef8

file case syntax fixed, added cred.yaml to deployment files, added ad…

de97392

…ditional tasks to do some of env fab script

checking in latest. Still getting an issue starting LO

817e36f

change to systemd

b466a1b

typo in default flock file.yaml

9c0b7d5

typo in default flock file.yaml

5033fb4

Fix case issue

1575b6b

pmitros force-pushed the devops_toy_sba branch from fd81ab9 to 1575b6b Compare September 30, 2024 14:25

pbrost and others added 2 commits September 30, 2024 10:31

added default flock creds.yaml

1cf9a25

python script to update hosts

6bcaa8e

pbrost added 3 commits September 30, 2024 15:25

fixed hosts.py

d9e269d

dynamic_hosts.py created

0c01041

fixed dynamic_hosts.py to work with scripts

5753b71


		def install_git_repos(ip):

		git_username = orchlib.config.creds.get('git_username')

               pyyaml
               fabric
+              unzip

Conversation

pmitros commented Sep 16, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants