Sharing your centralized location for audio plugins

[kmturley]

There seems to be a few centralized audio plugins locations including:
https://central.owlplug.com/store
https://getdunne.net/GyL/owl.php
https://getdunne.net/Krakli/owl.php
https://central.owlplug.com/mirrors/amvst/store
https://central.owlplug.com/mirrors/vst4free/store
https://www.kvraudio.com/plugins/vst-plugins/vst3-plugins/free/newest
https://github.com/VolantisDev/vst-guide-api/tree/master/plugins
https://github.com/joereynolds/reap-get/blob/master/src/data/packages.json

I have also made my own here:
https://github.com/kmturley/audio-project-manager/blob/master/lib/data/registry.json

I see you've started to standardize the json data format, and find a way to host plugins so they can be downloaded directly (without websites). I was wondering if others can connect to your centralized location too?

For example my tool is trying to replicate npm install so that a music project contains a project.json with all the plugins:
https://github.com/kmturley/audio-project-manager

Rather than me creating my own repo of plugins, should we start sharing? I have structured my json in a way that you can look up a plugin by id quickly:

"1208-reviber-by-1208-audio": {
    "author": "1208 Audio",
    "description": "A Musical Two Knob Audio Enhancer",
    "homepage": "https://1208audio.com/downloads/reviber",
    "id": "1208-reviber-by-1208-audio",
    "media": [
      "https://static.kvraudio.com/i/b/reviber.1589061770.png"
    ],
    "name": "1208 Reviber",
    "sources": {
      "darwin": "https://1208audio.com/downloads/reviber",
      "win32": "https://1208audio.com/downloads/reviber",
      "win64": "https://1208audio.com/downloads/reviber"
    },
    "tags": [
      "Exciter / Enhancer",
      "Equalizer",
      "Mixing and Mastering",
      "Tone Shaper"
    ],
    "version": "1.4.1"
}

[DropSnorz]

Hi @kmturley ! Very cool project.
OwlPlug users asked me for a way to install audio plugins using a CLI. They will be happy to see this project.

Of course, you can connect a third-party tool to any collection hosted on central.owlplug.com/*.
If you are using one of these endpoints, don't hesitate to let me known, so I can send you a message in case of breaking changes in the specification.

OwlPlug Store Specification has some lacks:

• Plugin identification. There is no id. I've recently added the Product.technicalId which can be bound to Audio Plugin UID following the VST Specification.
• Multi-version support. Only one version (semver) can be distributed. For example, we can't have the 1.0.0 and 1.0.1 in the same collection.

I want to support interoperability initiatives. We can create a new standard specification used by both tools:

• Any other tool can follow this spec to use our repositories. (Owlplug, apm, reap-get, VSTManager,... )
• All creators, developers, distributors, websites can set up a unique API to distribute plugins with different tools.

Maybe something like: AUdio Plugin Repository Open Specification (AUPR OS)

To be honest, I'm quite hyped by the idea. If you want to join the party with me, we can collaborate to build this common specification together. 🙂

[kmturley]

Yes I would be interested! I think the music production world is lacking a standardized format. The more people we can get to agree to a format, the better! I can tell you a little bit about my approach:

Package manager pattern
My approach was to replicate package managers as closely as possible. Since they have spent many years refining the commands/structure, and the benefits are clear:

• lists the packages your project depends on
• specifies versions of a package that your project can use using semantic versioning rules
• makes your project reproducible, and therefore easier to share with others

There are lots already existing:

• JavaScript has npm
• Python has pip
• Java has maven
• Ruby has gems
• Php has composer
  etc..

I was trying to follow npm as closely as possible they use a package.json:
https://github.com/kmturley/audio-project-manager/blob/master/package.json

I turned it into a project.json
https://github.com/kmturley/audio-project-manager/blob/master/example/Test%20Project/project.json

NPM has a registry search:
https://registry.npmjs.com/-/v1/search?text=commander&size=20

And then a registry result containing multiple versions:
https://registry.npmjs.org/commander

I turned it into this:
https://github.com/kmturley/audio-project-manager/blob/master/lib/data/registry.json

I changed the json list from an Array to Object to allow faster lookup of a package by id:
registry[id]

[DropSnorz]

I agree, we should create something similar to how package manager indexes work. Most developers are already familiar with this workflow. I like how npm is designed so we can keep base ideas. We must agree on a common terminology to identify elements. For our top-level component, it seems registry is better than repository.

If I understand right, both formats (Owlplug Store and apm registry) works as a "flat file" mode or unique API endpoint. So the complete registry is available in a single HTTP call. In this scenario, all searches and lookups are performed on client-side. We can keep this behavior as it's pretty simple to implement for any registry providers.

From my point of view the biggest challenges will be:

• How to identify plugins with a unique property. Conventions to attribute plugins id? (implemented in apm registry)
• Support multiple plugin formats VST, VST3, AU, ... (implemented in Owlplug Store)
• Support multiple versions for the same plugin (Not implemented in both formats)

I propose the following actions:

• I will create OwlPlug/plugin-registry-specification or OwlPlug/audio-plugin-registry idk. I think we can host the spec on the OwlPlug organization page, so it's not strongly coupled with our personal Github accounts. Of course, maintainer rights will be granted to you.

• I will probably move this issue to the new repository to keep our technical discussion.

• We can draft the spec with core concepts. I will have a very limited amount of time in June, but one of us can start by init basic files and concepts on a PR.

Are you ok with this?

[kmturley]

registry is better than repository.

Yep agree. Maybe we can setup https://registry.owlplug.com/ to test this new version? and if it works nicely, you can gradually migrate OWLPlug software over to it?

How to identify plugins with a unique property. Conventions to attribute plugins id? (implemented in apm registry)

npm works by having unique slugified ids (lowercase with hyphens) I like because they a human readable and easy to remember. And if someone has already taken the slug/id you can use a different variation.

Support multiple plugin formats VST, VST3, AU, ... (implemented in Owlplug Store)

I would guess, that the end-user does not care about the plugin format, they care more than the plugin will work on their computer? win32, win64, macos? maybe we can install all the formats which are compatible with their desktop computer? and let them choose which to use inside their music software?

I started to implement it like this:

"sources": {
      "darwin": "http://static.kvraudio.com/files/3653/dystroyed_v1-0-4_win.zip",
      "win32": "http://static.kvraudio.com/files/3653/dystroyed_v1-0-4_win.zip",
      "win64": "http://static.kvraudio.com/files/3653/dystroyed_v1-0-4_win.zip"
},

As it's an object you can reference the correct binary in NodeJS using process.platform:
sources[process.platform]
https://nodejs.org/api/process.html#process_process_platform

I looked at some npm packages which require platform binaries and couldn't seen a difference in the npm code. This is one for video editing: https://registry.npmjs.org/ffmpeg-static if you look at the code:

https://github.com/eugeneware/ffmpeg-static/releases/download/${release}/${platform}-${arch}
https://github.com/eugeneware/ffmpeg-static/blob/master/install.js#L141

it is just download a binary from the releases automatically:

https://github.com/eugeneware/ffmpeg-static/releases/

This is nice because the releases can be updated, without needing to update the registry json.

Support multiple versions for the same plugin (Not implemented in both formats)

Again broken record here, npm has already solved versioning for us too. They use an object:

so you can look up a specific version for you platform using:
versions[version].sources[process.platform]

They also have latest as a version which points to the latest version as a shortcut. So if you do:
npm install fmpeg-static

it will default to latest but you can pass a specific version using the @ sign:
npm install fmpeg-static@4.2.3

I will create OwlPlug/plugin-registry-specification or OwlPlug/audio-plugin-registry idk. I think we can host the spec on the OwlPlug organization page, so it's not strongly coupled with our personal Github accounts. Of course, maintainer rights will be granted to you.

I think call it OwlPlug/registry we can define the spec and write the code for the api exactly how you've done with https://central.owlplug.com/store?

I will probably move this issue to the new repository to keep our technical discussion.

Sounds good

We can draft the spec with core concepts. I will have a very limited amount of time in June, but one of us can start by init basic files and concepts on a PR.

Yeah no worries, we are all busy, we can do this slowly. In the meantime I can try and hook up the apm tool to the existing central store to see how it works!

One other thing:
You might want to migrate https://github.com/DropSnorz/OwlPlug over to the organization too?
I have some ideas, rather than coding it in Java, you could use https://www.electronjs.org/ which would be easier to publish and maintain compatibility. (but that's something for later!)

[kmturley]

I've made a branch of the apm command line tool which connects to OwlPlug central registry.
https://github.com/kmturley/audio-project-manager/tree/feature/owlplug-store

It works! But if you check the code diff before and after, you'll see that I had to add more loops to look up the package name, and bundle. This is fast now, but as more packages get added it won't scale very well!
studiorack/studiorack-cli@a36bcd9

[kmturley]

I've started to detail all of the necessary components of the ideal experience. (There are a lot!)
https://github.com/kmturley/audio-project-manager/wiki

This is a work in progress, but i'm going to try and create a prototype of this ecosystem e.g.

• plugin, which is automatically deployed as a release on github releases
• registry which has a scheduled job which looks for new versions and publishes a json feed to github pages
• website which is statically generated from the registry json
• command line tool which downloads the registry json, and plugin source
• app which connects to the registry json
• project containing plugins and can be updated using the tools

This is a pretty ambitious list, but I think if I can get an end-to-end prototype working, we can use that to align on a shared vision for this.

[DropSnorz]

Yep agree. Maybe we can setup https://registry.owlplug.com/ to test this new version? and if it works nicely, you can gradually migrate OWLPlug software over to it?

I really like the idea of automated registry updates based on GitHub releases. I can try to set up a quick POC for the OwlPlug registry as its dedicated to open-sourced plugins.
This will only work if developers are using Github releases which is not always the case. So, for the first prototype, some binaries will be downloaded from GitHub and some others will stay on OwlPlug Central servers until developers want to migrate to a GitHub release flow.

npm works by having unique slugified ids (lowercase with hyphens) I like because they a human readable and easy to remember.

Yes, I thought about it, using ids and an "associative array" make sense for something called registry. 🙂

I would guess, that the end-user does not care about the plugin format, they care more than the plugin will work on their computer? win32, win64, macOS? maybe we can install all the formats which are compatible with their desktop computer? and let them choose which to use inside their music software?

It depends on how it's used by third-party apps. In some cases, users may want to install only one type of plugins.

For APM use cases, plugins are scoped per-project but for OwlPlug, plugins are installed globally on the system. The end-user may want to install their plugin in recommended folders or custom ones.

VST
/Library/Audio/Plug-Ins/VST

Audio Units
/Library/Audio/Plug-Ins/Components

In early versions, OwlPlug was developed to install all plugins in a root directory. I received lots of feedbacks from users who want to keep their plugins installed in default folders. That's why OwlPlug binaries (bundles) are isolated by plugin format:

"bundles": [
  {
    "name": "Win64 VST",
    "targets": ["win64"],
    "format": "vst",
    "downloadUrl": "https://central.owlplug.com/products/squeezer/bundles/squeezer-w64-vst2_2.5.4.zip",
    "fileSize": 4500000
  },
  {
    "name": "Win64 VST3",
    "targets": ["win64"],
    "format": "vst3",
    "downloadUrl": "https://central.owlplug.com/products/squeezer/bundles/squeezer-w64-vst3_2.5.4.zip",
    "fileSize": 2600000
  }
]

We can imagine a sort of -g option for apm. This could install plugins in custom directories if configured. It may be great for command-line lovers that want to keep plugins at the system level.

If we want to support this use case, I see two solutions:

• A fat archive with all plugins formats inside. (Maybe with a normalized file structure to post-process it after the download)
• 1 archive = 1 format for 1 distrib.

Anyway, I think we should add the information of available types even if binaries are packaged in a single archive. This info may be relevant to display in a UI (Portal, Apps....)

apm install -g your-plugin

npm has already solved versioning for us too.

Yes, we can apply the same strategy used by npm.

You might want to migrate https://github.com/DropSnorz/OwlPlug over to the organization too?

Yes, I will probably transfer the repository to the OwlPlug organization in the future.

I have some ideas, rather than coding it in Java, you could use https://www.electronjs.org/ which would be easier to publish and maintain compatibility. (but that's something for later!)

Totally agree. I have this in mind for a few months. It's a huge work btw. I've started this project in 2017 as a quick draft during my IT studies. I've received a lot of feature suggestions that will require some changes to core architecture... And I won't implement these changes in the Java client. Moreover, Javascript is in trendings and it might help to build a contributors community.

I've just seen that you've created the audio-project-manager org. I'll take a look in the next days.