How is looping handled for multiple groups of speakers with differing lengths?

[hburgund]

@shreyas-jadhav How does the current looping playback system handle multiple "choirs"? A "choir" can be defined as a group of speakers that are related to each other. By definition, they will all have the exact same length of audio and should be played back in a looping fashion together.
When using this playback method, there will almost always be multiple choirs that exist in different physical places, but will be adjacent and could overlap each other in some cases.
Does the current looping playback system handle this multiplicity of choirs potential? I assume in order to handle this, it would have to use the speaker relationships to determine which speakers to group together? Or maybe it doesn't really matter and speakers just loop based on their length and because all speakers in the same choir are exactly the same length, all is fine?
I may be over-thinking this, just wanted to note the question so I can hear your thoughts since it is extremely important that multiple choirs of different lengths can be handled within the same project.
LMK!

[shreyas-jadhav]

Currently, there is no provision to handle different-length groups of speakers.
So all the speakers just start at same time, and loop whenever their individual end is reached, regardless of which choir they belong to.

But i can add this. But i'm not very sure how to proceed,
Suppose all the groups start at same time (or played somehow that it feels like they were started at same time) to ensure synchronisation,
now should the shorter group wait until the loop point of larger group is reached?

[hburgund]

Thx for this explanation of how things work now. I realize it is getting a bit complicated and I haven't explained it all fully, so I made a diagram to help me gather everything together and explain it to you.

I have been testing the simplest case which is the Choir B situation where there is one set of speakers all of which are the same exact length having been recorded using the looping recording method. As Listener Z moves around within these speakers, we get a nice evolving set of loops playing per the new playback method and everything seems to be looping and syncing nicely.

But things get complicated when we look at Listener Y. Though the choirs will initially start separated from each other geographically, as participants add their own voices, those new speakers will extend the range of the choir, but also, we have the mechanism that extends the speaker range for each speaker from which a new speaker is recorded. This is all to say that over time, the spaces between the choirs will diminish and eventually, we will get overlaps. This is by design and we hope it will create interesting musical situations.

I think that if we were to start all speaker loops at the same time and just have them playing a volume=0 when not active, then we would be OK just letting all speakers loop whatever their length is and we wouldn't have to worry about whether a speaker was part of Choir A or B etc. But since we will have so many speakers and we don't want to waste resources playing them all constantly (#83), things get a bit complicated.

In the progressive loading situation - based on proximity as determined by the speaker loading range indicated on the diagram - if a listener walks from position X to position Y, they will initially have only Choir C loops loaded all which will have the same 17s loop time, but as they get closer to Choir A, speakers from Choir A will need to be loaded and those have a different loop time. It will be important that the first Choir A speaker to start playing starts at the loop point of Choir C (which is already playing) because we will want the overlap regions to be connected by this shared start time. And the other critical things is that all other Choir A speakers that are added subsequently are started at the same Choir A loop point (which we know will no longer be shared with Choir B due to the differing lengths). Using the speaker relationships, this shouldn't be hard to determine which loop point should be used for which new speaker, but is critical.

@shreyas-jadhav LMK what you think about this and if it makes sense! Happy to hear all your thoughts on how best to accomplish this.

[hburgund]

Not sure if this would help, but we could consider keeping the original seed speakers all loaded all the time which would give additional speakers something to latch onto. Maybe this wouldn't help, but there won't be too many seed speakers, just one per choir, so even in larger installations, there would be 10-20, not hundreds, making this more feasible if it would help.

[hburgund]

Suppose all the groups start at same time (or played somehow that it feels like they were started at same time) to ensure synchronisation, now should the shorter group wait until the loop point of larger group is reached?

This is an important question and I tried to address it above, but will explain further. Basically, the answer is No; each speaker should loop (without a pause) at the length of its audio file. But there is some nuance. We want each choir to essentially be looping independently of all other choirs, but we want the start of each choir to be sync'd.

I'll make another diagram to explain, but the different choirs will likely have tempo similarities, so even if they are different numbers of measures/length, they will share tempo, so we want them to start at the same time so overlapping regions don't sound totally wacko with offset beats.

[shreyas-jadhav]

OK, I think i understood most of the part.

some thoughts from implementation POV

i don't reckon this would be most correct approach,
but based on how speaker engine is structured currently, here is what i imagine we could do...

Setup:

1. for each speaker, associate it with it's parent (or actually the oldest ancestor) speaker ID, if no parent, assign itself.
   now we can differentiate between groups. basically we can say parent_id as group_id
2. create a Map, with key as group_id and value as a time, initially all time values are null.

Loop Point Algorithm:

run the selection algorithm normally

1. if the selected speaker is of a group that is never played ever since mixer started, (i.e value in map is null)

   • set the value of that group_id as Date.now()
   • play the speaker immediately (i.e on the loop point)

2. if the selected speaker is of a group that was already played before,

   • Don't play immediately, but schedule a start in (duration - (currentTime - groupStartedAt) % duration) seconds

The above formula basically should calculate (if i'm correct),
how much time is needed for the future loop point of its group to come.

I think this should follow the behaviour that you specified.

for example,

• assume time starts at 0

• loop points of Choir C are
  0,17,34,68...

• at loop point 17 user was reached at intersection,

• now the assume a speaker (speaker A.1) for Choir A was selected by the algo,

• speaker A.1 start immediately at 17, (because this group was never player)
  the loop points of Choir A group become
  17,37,57...

• assume base speaker is still from Choir B, and at the loop point 34,
  assume the speaker A.2 is now selected, then it will not start immediately this time, but will be scheduled to start in 3 seconds i.e when time 37 will come

LMK if you think this would work to achieve what you are suggesting.

[hburgund]

Maybe this will help as well :-) This describes a theoretical listener journey related to the above 3-chorus diagram.

[hburgund]

I believe what you've outlined above will do the trick. LMK if my diagram reinforces your understanding or if it brings up more questions! I'm working this stuff out in my head as I go and there can certainly be some flexibility if that helps from a dev POV, so lmk if you have any ideas.

One thing I hadn't thought about is the base loop when in locations where multiple choirs are overlapping, but I guess maybe this will just get handled by the standard algorithm behavior? The base loop is set as the top available ancestor of the currently available speakers, right? (if multiple exist, take the one with the larger volume setting) But if there are two hierarchies, one for each choir, how will this work? I'm fine with doing something for now like keep the base loop with the current choir until there are no speakers left available for that choir and then take the base loop from the available choir per the standard selection algo. Trying to make this simple; lmk what you think!

Also, please keep in mind that we can setup and restrict things in certain ways if that would solve difficult technical issues or just make the code more efficient and streamlined. We don't need to build for every possible situation, in other words. I could say, for example, that we will not allow for more than 2 or 3 choirs to ever overlap. Not sure exactly how we would enforce this, but you get the point.

[shreyas-jadhav]

Doubt:
There will always be a single base loop at a time, right? For example, if any other loop comes (with no parent available) and it is closer, then it will start calling "loop algorithm" on every loop point, and previous base speaker will no longer call the loop algorithm. but it can pick the another group's base loop sometimes.
From the diagram above it loops like there are 2 base loops constantly playing, picking their children independently & simultaneously.

[hburgund]

Doubt: There will always be a single base loop at a time, right? For example, if any other loop comes (with no parent available) and it is closer, then it will start calling "loop algorithm" on every loop point, and previous base speaker will no longer call the loop algorithm. but it can pick the another group's base loop sometimes. From the diagram above it looks like there are 2 base loops constantly playing, picking their children independently & simultaneously.

Good point! I've updated the diagram, but does this one make sense now?

[shreyas-jadhav]

take the one with the larger volume setting

Now, that i think, i think instead of volume, we should prioritise over something like distance to center of mass. might be off for certain polygons with weird shapes, but should be more relevant in most cases. in my opinion.

[hburgund]

Now, that i think, i think instead of volume, we should prioritise over something like center of mass. might be off for certain polygons with weird shapes, but should be more relevant in most cases. in my opinion.

Good idea. Let's do that. I think those calcs are simple with turf yes? Again, don't want to make things vastly more complex, but if this is an easily accessible calc, then let's do this as I agree is is more sensible.