Update gameobject tools

[msanatan]

LLMs tend to get tripped up by some basic functions, and I believe a re-org of the tools could help make the LLMs select the right things more quickly, and we can give the users more capabilities:

manage_gameobject currently:

Retrieves:

• GameObject Basic Info
  • Instance ID, name, tag, layer
  • Active state, static flags
  • Transform data (position, rotation, scale)
  • Parent/child hierarchy relationships
  • Prefab status and connections
• Component Data
  • All components on a GameObject (get_components)
  • Single component data (get_component)
  • Serialized component properties (public + [SerializeField] private fields)
  • Component instance IDs and type names
• Search/Find Results
  • Find by: ID, name, path, tag, layer, component type
  • Support for finding in children, inactive objects
  • Returns full GameObject data for found objects

Updates:

• GameObject CRUD
  • Create: Empty, primitives, or prefab instances
  • Modify: Name, parent, tag, layer, active state, transform
  • Delete: Single or multiple GameObjects
• Component Management
  • Add: Components with optional initial properties
  • Remove: Components by type name
  • Set Properties: Nested property support (e.g., material.color, materials[0].color)
• Special Features
  • Prefab creation and instantiation
  • Undo/Redo support throughout
  • Physics component conflict detection (2D vs 3D)
  • Tag/Layer auto-creation
  • GameObject/Component reference resolution

There are many problems with how it's structured, here are some of the important ones:

• LLMs trip themselves up with parameters, there's literally too much for this tool.
  • These are pretty basic operations and it annoys me that it hast to make 2/3 calls to get things right
• We can't see what components are in the GameObject w/o serializing everything
• The ManageGameObject.cs is 2.5K lines long, and really hard to test and maintain
• There's a lot of redundant serializing in ManageGameObject's logic, and it happens twice because of how we list the scene hierarchy

So I have a solution for this:

• Tools
  • manage_gameobject - GameObject lifecycle (create, modify, delete)
  • manage_components - Component lifecycle (add, remove, set_property)
  • manage_scene - Scene lifecycle (create, load, save)
  • find_gameobjects - Search operations (returns IDs only)
• Resources
  • unity://scene/active - Active scene info
  • unity://scene/active/hierarchy - Scene hierarchy with component type lists
  • unity://scene/build-settings - Build settings scenes
  • unity://gameobject/{instanceID} - Single GameObject info (no components)
  • unity://gameobject/{instanceID}/components - All components with full data
  • unity://gameobject/{instanceID}/component/{name} - Single component data
    • There's a possibility a component can have the same name, so we may do index number

Benefits:

• Less confusion for the LLMs using the tools
• Easier to manage development wise
• More data of how this plugin is actually used

[Scriptwonder]

I agree with many of the existing issues, but I feel there should be a better plan than the four tools you currently plan (I am thinking for future PRs such as batching/multi-thread, which I am currently working on)? Let me gather more thoughts and edit my response, but this is a needed PR!

[msanatan]

Aside from the somewhat related issues around optimising this tool, batching is one of the reasons I'd like to do this. I believe only deleting gameobjects is supported. I'm glad to hear you're already working on it 🙂

[dsarno]

I agree, manage_gameobject is way bloated. It's also the most popular tool by far.

My first PR way back in April was for serializing component data. It's a useful feature but it wasn't written with good architecture in mind. Due for improvement and single responsibility focusing.

I can help with this too. We need to add a notion of "get/read" to @msanatan's plan -- getting data is super helpful for debugging in Unity.

But do we do this before or after we figure out if we have to rewrite the server in C#?

[dsarno]

I guess the read part is from resources. Which I've tried to implement in the past but most clients didn't handle until recently. So it'll be good to take another look at how to implement those like you say.

[msanatan]

I guess the read part is from resources. Which I've tried to implement in the past but most clients didn't handle until recently. So it'll be good to take another look at how to implement those like you say.

Yea, that's one thing about MCP - if you read the spec and test in Claude, you'd think everything works great but that's not case for other clients. I use VS Code as my main barometer because it's super popular with our users, and somehow, technology wise they're a bit behind the others haha.

[msanatan]

@dsarno I think we won't go the C# route, at least soon. If we do it in dotnet, then we have the same problem. We can run the MCP server outside of Unity's main thread, but it's a considerable refactor.

Right now, I'm hoping to host the server remotely, with some session IDs that identify a user, and have users connect. In this way, they won't even need python or uv, they'll just need the plugin and their IDE.

[dsarno]

@msanatan But doesn't that conflict with it being open source? It means the server becomes effectively sole source and proprietary. It would be harder for others to develop and debug.

I know we want to be on the asset store, but I'm not sure if that should be the tail wagging the dog for the structure of the repo

[msanatan]

Why would it no longer be open source? It's the exact same MCP server. Most users will connect to localhost, and those on the asset store will default to a hosted version. The server's code won't change or be locked away

[dsarno]

I see, I didn't understand that the idea that GitHub repo would still point to local by default. So you would have some kind of mirror for an asset store version that points to the remote server? Or it sets an env var to remote?

I think there's still a risk with having the mcp or asset store version depend on a proprietary server. It means someone has to be in charge of maintaining the server, which again feels not very "open" and community accessible. Happy to talk more offline if it helps.

[msanatan]

Happy to talk here in case others have similar concerns. Right now we kind of dictate how the user connects, ideally it's something they can fully configure - including if they'd like HTTP/stdio (I put the ports in text fields in the UI to prep for that).

By default users point to a local network. On the asset store, they'll point to a remote URL by default. Installing Python + uv would allow them to use a local server instead. They're not locked in. It's also going to be the exact same MCP server being built by the Dockerfile, so any URL is interchangeable.

Can you elaborate on what are the particular risks of Coplay hosting an MCP server? Particularly when running it locally is an option, as well as anyone else hosting it remotely as well?

[dsarno]

Well there's the scenario where the server goes down and Coplay team is busy or unavailable. Then everyone using the asset store version is affected, and may not know why, or be able to set up localhost as a fallback.

In a more unfortunate scenario where Coplay goes away completely, everyone has to figure out how to fix the store asset to work on a different server (maintained by who?) or revert to the situation where the asset has to be rewritten to work without a remote server.

It may be a question of whether the unity store asset is effectively a Coplay-owned product, (which is fine since the repo is open-source), or if it is meant to be a community owned extension of the repo. If the latter, it seems like the asset should not rely on systems the community does not control.

But I think this is a partially moot conversation if the OSS repo continues to point to local by default, and not depend on there being a remote server. If we really need to figure out how to fix an issue with the asset store version later, we can.