diff --git a/README.md b/README.md index c35395b..6a6f67d 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,10 @@ # devops.el: Org-based devops -This package allows you to provision and manage infra +This package allows you to provision and manage infra using org mode notebooks ## Requirements - magit -- agent shell -- Claude Code ## Installing @@ -16,15 +14,18 @@ This package allows you to provision and manage infra :vc (:url "https://github.com/unifica-ai/devops.el")) ``` -## tools.org — Per-project Library of Babel +## devops-lob Any project with a `tools.org` at its root can expose named org-babel blocks as reusable tools. `devops-lob` loads and tracks these per-project so they don't pollute other projects. ### Setup ```elisp -(require 'devops-lob) -(devops-lob-auto-mode 1) ; auto-load on find-file +(use-package devops + :ensure t + :vc (:url "https://github.com/unifica-ai/devops.el") + :hook + (after-init . (lambda () (devops-lob-auto-mode 1)))) ``` With `devops-lob-auto-mode` enabled, opening any file in a project that has `tools.org` automatically loads its named blocks into the org-babel Library of Babel. diff --git a/decisions/01-tangling.org b/decisions/01-tangling.org new file mode 100644 index 0000000..6e8f217 --- /dev/null +++ b/decisions/01-tangling.org @@ -0,0 +1,183 @@ +#+TITLE: Decision 1: Devops-tangle command +#+DATE: 2026-05-08 +#+LINK: noweb https://orgmode.org/manual/Noweb-Reference-Syntax.html +#+LINK: org-babel-langauges https://orgmode.org/worg/org-contrib/babel/languages/index.html + +* Context + +Emacs can "tangle" org-mode source blocks to files. When tangling a file, +emacs reads through each source code block, and if it has a =:tangle= header +argument, it extracts it to the a file. For example, calling =org-babel-tangle= +in a buffer with the following source block would create a file called =config.json=: + +#+begin_example +#+begin_src json :tangle=config.json +{ + "name": "Frodo" +} +#+end_src +#+end_example + +The core Babel functions (viewing, export, tangling, etc.) are language agnostic and will work even for languages that are not [[org-babel-langauges][explicitly supported]]. In the example above, even though we can't execute =json= blocks, we can still tangle them. + +A lot of nice things fall out of having a single document as the source of truth. +The entire context is in one file, together with explanatory notes. +This is great for people, and great for AI agents. + +By passing a TRAMP remote location to the =:tangle= parameter, we could use this +mechanism to maintain server configuration, tangling files to many servers arbitrarily. + +However, there are some issues with doing this. + +First, it's verbose and not DRY. The same servers would show up in many paths, as long-ish connection strings. + +Second, emacs tangles the entire buffer at once[fn:1], with all its source code blocks. If some =:tangle= locations point to local files, and others to (maybe several) servers, things will get messy. + +[fn:1] You can narrow the buffer to tangle only part of a file, perhaps with =org-narrow-to-subtree= which improves matters somewhat.) + + +Third, it's not clear how we upload the same configuration to many machines (a common devops task). + +And most problematically, if something goes wrong - as it might with remote destinations rather than local - +it may not be obvious which tangling operations were successful and which not, leaving servers in an unknown state. + +Out of the box, tangling does /almost/ everything we need, but it does not impose enough structure or oder. +We want to implement some conventions on top of tangling. + +* Decisions + +** 1. Custom tangle command + +We could avoid a custom command by adding "advice" the =org-babel-tangle= command, so users would have nothing new to remember. + +However, + +1. It would be hard to circumvent the file scope difficulties mentioned above. If by default I want to + apply all configuration changes to a machine, i.e. tangle several related files in one go, + I would have a lot to compute in the advice function, and it would be hard to understand + what was executed and what was not. + +2. It would rely in a lot of temporary, invisible, mutable state, and a lot of variable + rewriting on the fly, making it brittle. + +Probably better to implement a =devops-tangle= command, rather than advice on =org-babel-tangle=. Roughly, this command should + +1. Copy entire buffer to a temp buffer +2. Resolve =:dir= from heading's server tag +3. Rewrite =:tangle= to full TRAMP path (=:tangle ~/foo.txt= to =:tangle /ssh:user@server:~/foo.txt=) +4. Narrow temp buffer to heading +5. Call =org-babel-tangle= + +A full buffer copy ensures =:var= refs to named blocks outside heading resolve +correctly. (Even if we're narrowing and only executing the current heading.) + +** 2. Use =:tangle= header + +Which header argument should =devops-tangle= command use to determine the target server? + +It could use =#+NAME=. If =NAME= is a file, then add =:dir= to it and output =:tangle=. +For instance, this might upload =foo.txt= to the =server1=: + +#+begin_example + + * Foo :server1: + + #+NAME: ~/foo.txt + #+begin_src + foo contents + #+end_src + +#+end_example + +The format is nice: looks like a figure and its caption. However, itt makes global name collisions likely: If we are uploading files with the same names to different servers,multiple source code blocks would have the same name. + +As an alternative, we could have a custom variable like =:target=. It would require both a magic variable and a custom tangling function, +making this inelegant. Also, it may break some intuitions people already have built up with =org-babel-tangle= +locally. + +It seems best to use the standard header: =:tangle ~/foo.txt=, and the existing tangling algorithm, +which already uses this header. + +** 3. Heading is default scope + +The default tangle functionality is either too small (one source code block); or too big (entire file), +with a somewhat mitigating feature in footnote 1. + +The devops.el package takes the convention that one heading = 1+ labeled servers. It's easier to see +at a glance what is running where. + +Matching this convention: + +- =devops-tangle= should tangle current heading / server / servers. +- =C-u devops-tangle= should tangle the entire buffer + +** 4. Use Noweb for templating + +Some configuration blocks will have computed references, for things such as machine names and +secret references. + +We could try passing =vars= into source code blocks. However, these are not text replacements, but rather integrate into each target language. As such, they work only work in [[org-babel-langauges][executable languages]]; interpolation syntax varies (for instance, in shell scripts, variables are =$foo= in source blocks, while in Python they are =foo=; and the output file will contain some setup code that is not in the original source code block. + +We could implement custom templating system on top of jinja, like the following snippet hallucinated by Google: + +#+begin_example + +#+NAME: my_data + +| Name | Role | +|-------+-----------| +| Alice | Developer | +| Bob | Designer | + +#+BEGIN_SRC jinja :var data=my_data +Members of the team: +{% for row in data %} +- {{ row[0] }} is the {{ row[1] }} +{% endfor %} +#+END_SRC + +#+end_example + +Although it seems useful, this doesn't exist. + +Org mode already has [[noweb]], which can include one block textually in another, and even evaluate a source block before including it. We could use it like this to upload an env file with a password. + +#+begin_example + + * Local + + #+name: db-pass + #+begin_src elisp + (auth-source-pick-first-password :host "db") + #+end_src + + * Server :server1: + + #+begin_src env :tangle ~/db.env :noweb yes + DB_PASSWORD=<> + #+end_src + +#+end_example + +It seems noweb has enough features for basic templating. + +** 5. Fail fast on errors + +We are tangling multiple blocks at once. If one fails (e.g. TRAMP connection, permissions), we should stop +immediately. We should let the user know which block failed, and we should know that previous blocks under the same heading completed successfully. + +** 6. Tangle to multiple servers + +This should upload the same file to =server1= and =server2=: + +#+begin_example + + * Upload file :server1:server2: + + #+begin_src json + { + "name": "Frodo" + } + #+end_src + +#+end_example diff --git a/devops-test.el b/devops-test.el index d7aa949..32a22d4 100644 --- a/devops-test.el +++ b/devops-test.el @@ -50,7 +50,7 @@ (org-back-to-heading) (devops-set-header-args-from-tag) (should (equal (org-entry-get nil "header-args") - ":dir app@server1:")))) + ":dir /ssh:app@server1:")))) (ert-deftest devops-set-header-args-from-tag-no-match-test () "Error when no #+SERVER matches the heading's tags." @@ -64,4 +64,84 @@ (should-error (devops-set-header-args-from-tag) :type 'user-error))) +(ert-deftest devops--heading-server-tag-test () + "Find matching server tag on current heading." + (with-temp-buffer + (org-mode) + (insert "#+SERVER: server1 (source)\n" + "\n" + "* Deploy\t\t:source:\n") + (goto-char (point-max)) + (org-back-to-heading) + (should (equal (devops--heading-server-tag) + '("source" . "server1"))))) + +(ert-deftest devops--heading-server-tag-no-match-test () + "Return nil when heading tags don't match any server." + (with-temp-buffer + (org-mode) + (insert "#+SERVER: server1 (source)\n" + "\n" + "* Heading\t\t:other:\n") + (goto-char (point-max)) + (org-back-to-heading) + (should (null (devops--heading-server-tag))))) + +(ert-deftest devops--rewrite-tangle-paths-test () + "Rewrite :tangle paths to include TRAMP prefix." + (with-temp-buffer + (org-mode) + (insert "#+TITLE: Test\n" + "\n" + "* Heading\n" + "\n" + "#+begin_src sh :tangle ~/foo.txt\n" + "echo hello\n" + "#+end_src\n" + "\n" + "#+begin_src sh :tangle ~/bar.conf\n" + "key=val\n" + "#+end_src\n" + "\n" + "#+begin_src sh\n" + "echo no tangle\n" + "#+end_src\n") + (devops--rewrite-tangle-paths "/ssh:app@server1:") + (goto-char (point-min)) + (should (search-forward ":tangle /ssh:app@server1:~/foo.txt" nil t)) + (should (search-forward ":tangle /ssh:app@server1:~/bar.conf" nil t)) + ;; Block without :tangle untouched + (goto-char (point-min)) + (should-not (search-forward ":tangle /ssh:app@server1:no" nil t)))) + +(ert-deftest devops--rewrite-tangle-paths-skip-absolute-test () + "Don't rewrite :tangle paths that are already absolute." + (with-temp-buffer + (org-mode) + (insert "* Heading\n" + "\n" + "#+begin_src sh :tangle /ssh:app@server1:~/already.txt\n" + "echo hello\n" + "#+end_src\n") + (devops--rewrite-tangle-paths "/ssh:app@server1:") + (goto-char (point-min)) + ;; Should not double-prefix + (should-not (search-forward "/ssh:app@server1:/ssh:" nil t)))) + +(ert-deftest devops-tangle-no-server-tag-test () + "Error when current heading has no matching server tag." + (with-temp-buffer + (org-mode) + (insert "#+SERVER: server1 (source)\n" + "\n" + "* Heading\t\t:unknown:\n" + "\n" + "#+begin_src sh :tangle ~/foo.txt\n" + "echo hello\n" + "#+end_src\n") + (goto-char (point-max)) + (org-back-to-heading) + (should-error (devops-tangle) + :type 'user-error))) + (provide 'devops-test) diff --git a/devops.el b/devops.el index e14b14e..00c1522 100644 --- a/devops.el +++ b/devops.el @@ -38,42 +38,125 @@ (delq nil (mapcar #'devops--parse-server-keyword (devops-org-keywords "SERVER")))) -(defun devops-resolve-server-for-tag (tag) +(defun devops--resolve-server-for-tag (tag) "Look up TAG in #+SERVER keywords, return server name or nil." (cdr (assoc tag (devops-server-tag-alist)))) +(defun devops-resolve-server-for-tag (tag) + "Look up TAG in #+SERVER keywords, return server name or nil." + (devops--resolve-server-for-tag tag)) + (defun devops-set-header-args-from-tag () "Set :header-args: :dir from the current heading's tag and #+SERVER mappings." (interactive) (let* ((tags (org-get-tags nil t)) - (server (seq-some #'devops-resolve-server-for-tag tags))) + (server (seq-some #'devops--resolve-server-for-tag tags))) (if server - (org-entry-put nil "header-args" (format ":dir app@%s:" server)) + (org-entry-put nil "header-args" (format ":dir %s" server)) (user-error "No #+SERVER match for tags: %s" tags)))) -(defun devops--tangle-resolve-target () - "Return (FILEPATH . BODY) for the source block at point." - (let* ((element (org-element-at-point)) - (body (org-element-property :value element)) - (header (org-element-property :parameters element)) - (target (when (and header (string-match "target=\"\\([^\"]+\\)\"" header)) - (match-string 1 header))) - (info (org-babel-get-src-block-info 'light)) - (params (nth 2 info)) - (dir (cdr (assq :dir params)))) - (unless target - (user-error "No target variable found in source block header")) - (unless dir - (user-error "No :dir property found")) - (cons (concat dir target) body))) - -(defun devops-tangle-to-target () - "Tangle source block at point to :dir + target path." - (interactive) - (pcase-let ((`(,filepath . ,body) (devops--tangle-resolve-target))) - (write-region body nil filepath) - (message "Wrote %s" filepath))) +(defun devops--inject-dir-from-tag (orig-fn &optional arg info params) + "Advise org-babel-execute-src-block to inject :dir from nearest heading tag." + (let* ((tags (org-get-tags nil t)) + (server (seq-some #'devops--resolve-server-for-tag tags)) + (params (if server + (cons (cons :dir server) params) + params))) + (funcall orig-fn arg info params))) + +(advice-add 'org-babel-execute-src-block :around #'devops--inject-dir-from-tag) + +(defun devops--heading-server-tag () + "Return (TAG . SERVER) for the first matching server tag on current heading. +Searches heading's tags against #+SERVER keywords." + (let ((tags (org-get-tags nil t))) + (seq-some (lambda (tag) + (when-let* ((server (devops--resolve-server-for-tag tag))) + (cons tag server))) + tags))) + +(defun devops--rewrite-tangle-paths (tramp-prefix) + "Rewrite :tangle header args in buffer to include TRAMP-PREFIX. +Modifies buffer text. Skips :tangle no and already-absolute paths." + (save-excursion + (goto-char (point-max)) + (while (re-search-backward + ":tangle +\\([^ \t\n]+\\)" + nil t) + (let ((path (match-string 1))) + (when (and (not (string= path "no")) + (not (string-prefix-p "/" path))) + (replace-match (concat ":tangle " tramp-prefix path))))))) + +(defun devops--tangle-heading (source-buf heading-pos server tmp-file) + "Tangle HEADING-POS from SOURCE-BUF to SERVER using TMP-FILE as scratch. +Returns number of files tangled, or nil." + (let ((tmp-buf (find-file-noselect tmp-file))) + (unwind-protect + (with-current-buffer tmp-buf + (let ((inhibit-read-only t)) + (erase-buffer) + (insert-buffer-substring source-buf) + (goto-char heading-pos) + (org-narrow-to-subtree) + (let* ((files (progn + (devops--rewrite-tangle-paths server) + (org-babel-tangle)))) + (widen) + (when files (length files))))) + (with-current-buffer tmp-buf + (set-buffer-modified-p nil))))) + +;;;###autoload +(defun devops-tangle (&optional arg) + "Tangle current heading's source blocks to their remote server. +Resolves the heading's server tag to a TRAMP path and rewrites +:tangle header args before delegating to `org-babel-tangle'. +With prefix ARG, tangle all headings in the buffer that have +server tags." + (interactive "P") + (let ((source-buf (current-buffer)) + (results nil) + (tmp-file (make-temp-file "devops-tangle-" nil ".org"))) + (unwind-protect + (if arg + ;; Whole buffer: tangle each server-tagged heading + (org-map-entries + (lambda () + (when-let* ((pair (devops--heading-server-tag))) + (let* ((tag (car pair)) + (server (cdr pair)) + (heading-pos (point)) + (n (devops--tangle-heading + source-buf heading-pos server tmp-file))) + (when n + (push (list tag server n) results)))))) + ;; Single heading + (let ((pair (devops--heading-server-tag))) + (unless pair + (user-error "No #+SERVER match for tags on current heading")) + (let* ((tag (car pair)) + (server (cdr pair)) + (heading-pos (save-excursion + (org-back-to-heading t) + (point))) + (n (devops--tangle-heading + source-buf heading-pos server tmp-file))) + (when n + (push (list tag server n) results))))) + (when-let* ((buf (find-buffer-visiting tmp-file))) + (kill-buffer buf)) + (delete-file tmp-file)) + ;; Report + (if results + (message "%s" + (mapconcat + (lambda (r) + (format "Tangled %d file(s) to %s (%s)" + (nth 2 r) (nth 0 r) (nth 1 r))) + (nreverse results) "; ")) + (message "No files tangled")))) (defun devops-exec-in-notebook (block-name) "Execute BLOCK-NAME in the source deployment org file." @@ -160,7 +243,7 @@ BODY is the source block content to copy to clipboard." ((derived-mode-p 'dired-mode) (dired-current-directory)) ((derived-mode-p 'org-mode) - (when-let ((info (org-babel-get-src-block-info 'light))) + (when-let* ((info (org-babel-get-src-block-info 'light))) (let ((dir (cdr (assq :dir (nth 2 info))))) (when dir (expand-file-name dir))))) (buffer-file-name @@ -169,7 +252,7 @@ BODY is the source block content to copy to clipboard." (defun devops-src-block-env-vars () "Return alist of evaluated :var params from current src block." (when (derived-mode-p 'org-mode) - (when-let ((info (org-babel-get-src-block-info))) + (when-let* ((info (org-babel-get-src-block-info))) (let ((params (nth 2 info))) (delq nil (mapcar (lambda (p) @@ -186,7 +269,7 @@ BODY is the source block content to copy to clipboard." (defun devops-src-block-body () "Return the body of the current src block, or nil." (when (derived-mode-p 'org-mode) - (when-let ((info (org-babel-get-src-block-info 'light))) + (when-let* ((info (org-babel-get-src-block-info 'light))) (let ((body (org-trim (nth 1 info)))) (unless (string-empty-p body) body))))) diff --git a/examples/1_servers.org b/examples/1_servers.org new file mode 100644 index 0000000..d0de537 --- /dev/null +++ b/examples/1_servers.org @@ -0,0 +1,58 @@ +#+SERVER: /ssh:user@example.com: (source) +#+SERVER: /ssh:user@example.com: (target) +#+TITLE: Running server commands + +=devops.el= helps you run commands on one or more named servers by declaring them at the the top of the file in this format. + +#+begin_example +#+SERVER: server1.com (source) +#+SERVER: server2.com (target) +#+end_example + +Here, =server1.com= is the "source" server, and =server2.com= is the "target" server. There can be as many labeled =SERVER= blocks as we need. + +Normally, org-babel runs commands on servers whenever the =:dir= header argument +is set to a remote location. To run =ls= on =server1.com=, we would write + +#+begin_example +#+begin_src sh :dir /ssh:user@server1.com: +ls +#+end_src +#+end_example + +With =devops.el=, we can run the same command by adding a tag to the nearest heading: + +#+begin_example +** some-heading :source: + +#+begin_src + ls +#+end_src + +#+end_example + +* Try it + +Instructions +1. Modify the server blocks at the top of file (see caveats) +2. Run =devops-set-header-args-from-tag= +3. You should see header-args set as follows + +Try the same for =target= + +* List source :source: + +#+begin_src sh :results output +ls +#+end_src + +* List target :target: + +#+begin_src sh :results output +ls +#+end_src + + +* Caveats + +- You should be able to ssh into the server without a password diff --git a/examples/2_tangle.org b/examples/2_tangle.org new file mode 100644 index 0000000..b8d7ae1 --- /dev/null +++ b/examples/2_tangle.org @@ -0,0 +1,39 @@ +#+SERVER: /ssh:user@example.com: (app_server) +#+TITLE: Example 2: Tangle remote files + +=devops.el= helps you create files in remote servers. + +Source blocks with a =:tangle= header arg will be written to +the server resolved from the heading's tag. + +* Try it! + +** Upload my name :app_server: + +Use =devops-tangle= in this heading to upload =~/foo.txt= to the =app_server=. + +#+name: username +#+begin_src elisp +(user-full-name) +#+end_src + +#+name: emacs-version +#+begin_src elisp +(emacs-version) +#+end_src + +#+begin_src json :tangle ~/foo.json :noweb yes +{ + "name": "<>" +} +#+end_src + +#+begin_src txt :tangle ~/emacs.txt :noweb yes +<> +#+end_src + +When the command finishes, you should see this in the Messages buffer: + +: Tangled 2 file(s) to app_server (aly-2602-suite) + +The two files should have been uploaded to the server. diff --git a/examples/simple.org b/examples/simple.org deleted file mode 100644 index 101e61c..0000000 --- a/examples/simple.org +++ /dev/null @@ -1,44 +0,0 @@ -#+SERVER: server1 (source) -#+SERVER: server2 (target) -#+TITLE: Run commands on multiple servers - -Instructions -1. Place cursor in code block -2. Run =devops-set-header-args-from-tag= -3. You should see header-args set as follows - -#+begin_example -:PROPERTIES: -:header-args: :dir app@server1: -:END: -#+end_example - -Try the same for =target= - -#+begin_example -:PROPERTIES: -:header-args: :dir app@server2: -:END: -#+end_example - -* List source :source: - -#+begin_src sh - ls -#+end_src - -* List target :target: - -#+begin_src sh - ls -#+end_src - -* What's happening? - -=devops.el= parses the =SERVER= properties at the top of the file. - -#+begin_example -#+SERVER: server1 (source) -#+end_example - -It constructs the header arguments