Dependency Resolution and Custom Node Standards

Making ComfyUI more comfy.

Custom nodes aren’t just a part of ComfyUI — they’re at its core. This blog post is a first look at an initiative we’ve dubbed Nodes v3 — an attempt to improve the custom node experience in ComfyUI. This is still a work in progress, and we’re looking to get feedback from custom node authors on the parts you love, the parts you hate, and everything in between.

First, let’s talk about what the challenges we’re trying to solve with Nodes v3.

• Improve Stability — Today, every change made to ComfyUI has the potential to break custom nodes. Because there’s no separation between “internal” functions and a public interface, even the most well-coded extensions are subject to these issues.

• Solve Dependency Woes — Every custom node pack has its own set of dependencies. Unfortunately, sometimes these dependencies conflict which can “brick” a ComfyUI installation when you install the wrong custom nodes together.

• Dynamic Inputs/Outputs with first-class support - While custom nodes today can add arbitrary inputs and outputs on the frontend, some functionality doesn’t work with these dynamic inputs.

• Improved Custom Widgets — We want to make it easier for people to add their own widgets to nodes on the frontend and to send updates to those widgets while a node is executing on the backend. While these are both possible now, they aren’t easy and are fragile, frequently breaking with new ComfyUI changes.

• Streamline Model Management — New releases and custom nodes mean new models. Figuring out what models are already present locally, which need to be downloaded, and what folders to put them is a frustrating experience, especially as models grow in size.

• Enable Future Iteration — While these efforts aren’t yet in progress, we’re also looking to set up ComfyUI to be able to add the following functionality in the future:

  • Distributed execution of workflows across multiple machines.

  • Parallel execution of nodes on a single machine.

  • More!

The Plan

While we’re not quite ready to share all of our proposals for Nodes v3, here are some of them.

• To Improve Stability, we’re creating a Public API for custom nodes with both documentation and backward compatibility guarantees. While the long-term goal is to build out this API until custom nodes no longer need to hook into internal functions, we know that won’t be the case immediately and won’t be intentionally preventing nodes from continuing to work as they do today. Extensions that do only use the public APIs, however, will be protected from future changes.

• To Solve Dependency Woes, we’re exploring the ability to put each custom node pack utilizing the Public API in its own Python process. Even if we can’t get every extension isolated like that, reducing the number of custom node packs in the primary process reduces the chance of a dependency conflict exponentially.

• To enable first-class support for Dynamic Inputs/Outputs and Improved Custom Widgets, we’re formalizing a new schema for writing nodes specifically for v3. Currently, developers need to memorize many class methods and property names, special dictionary keys and the various modifiers accepted by specific inputs and widgets. Building on top of this pattern for dynamic behavior would make for an even tougher time for developers. The new schema will be more object-oriented and explicit, allowing developers to see exactly what built-in types are capable of and have access to sane type hints. We are also considering adding more flexibility to node definition, allowing inputs, widgets, and required/optional classification to be displayed in any chosen order.

Why Call It ‘v3’?

Internally, there was an effort a few months ago to fix bugs and restructure nodes on the frontend that was referenced as ‘v2’ with a corresponding RFC document, iterating on the original node schema that remained largely unchanged (‘v1’). Some of ‘v2’ got implemented, but its scope was small.

Since this is technically the third revision of the node specs, we went with ‘Nodes v3’.

Communication

While the design is worked on, the best place for feedback is our official Discord server. Specifically for custom node developers registered on Comfy Registry, we sent out invites to a private Discord channel (#exclusive-custom-node-devs) that will make it easy to communicate with the core development team. If you are a custom node developer and have not received an invite, join the Discord server and message a Comfy Org member or moderator!

Deep Dive

We’re ready to dive deeper into the details on a couple topics under this initiative. Keep an eye out for deep dives on the other topics in the coming weeks!

Node Specification

While this is still subject to change, v3 nodes schema will in general be more explicit and self-contained than its present form.

Consider these test nodes, written in the current and proposed schemas:

The V1 Test Node is written with the current schema:

Writing this node requires the definition of a total of 7 class methods or properties, describing the outputs of the node is done by providing tuples that correspond to the length of the total outputs, certain parameters must be declared as valid tuples (inputs and returned output), and two dictionaries are expected to be declared in the nodes file for both the class mapping and display name mapping.

Experienced ComfyUI developers have grown accustomed to this system, but much of available functionality is obscured by the dictionary structure. If you are a node developer reading this, were you aware you could have an integer widget become a slider by setting a ‘display’ key with the value of ‘slider’?

Here is the proposed v3 schema equivalent (subject to change at any time during development):

Inputs and outputs are all declared as objects in a list, with all possible parameters declared within a single class method: DEFINE_SCHEMA. The Schema object keeps track of all possible expected parameters, including node id and display name. Output parameters are all stored together instead of in separate lists, and are mandated to have an associated id, just like inputs, to allow for advanced dynamic behavior.

One key difference with v3 nodes is that the class interface is intended to be stateless - the execute function is a class method (receives ‘cls’) instead of an instance method (receives ‘self’). There will still be an object provided on the ‘cls’ parameter to act as an API for caching parameters as one would with the current interface. The reasoning behind this change is to allow custom node code to work interchangeably between unique environments - whether the code is executed on isolated processes or even separate machines, nodes will be able to not worry about what is going on under the hood.

Curated API for Backward Compatibility

There are a few specific goals for the creation of our node public API:

1. Prevent custom nodes from breaking when we update ComfyUI — and the corollary — let us refactor and improve ComfyUI without breaking all custom nodes.

2. Make it easier to develop and maintain custom nodes by having a documented API.

3. Provide a mechanism for extracting node execution from the main process to enable parallel execution, dependency isolation, and improve security.

First, here’s what using the API in an extension/custom node looks like:

Use by Custom Nodes

from comfy_api.v0_0_3 import ComfyAPI

api = ComfyAPI()

# ...
async def execute(images: ImageInput):
	b, h, w, c = images.shape
	for i in range(b):
		doThing(images[i])
		await api.set_progress(value=i, total=b, preview=images[i])
		# API Use ^^^^^^^^^^^^

There are a couple things you might notice.

Versioned API Objects

When importing the API for usage, you’ll specify a specific API version. Once a specific version is marked as “stable”, nodes using that version will continue to work even as we continue to grow and improve ComfyUI. In addition to the “version adapters” allowing old nodes to continue running without regular updates, those adapters will also serve as a guide for people looking to update to newer versions of the API in order to take advantage of newer features.

Version Lifecycle - In Development

At all times, we will have one version number that is under development. This version will not be stable and is subject to change. Generally, extension authors should only use under development versions if they need to take advantage of new functionality.

At a regular cadence (tentatively once a month), the development version will be moved to “stable” status and development will begin on the next version.

# comfy_api/latest/__init__.py
# All development happens in this module
class ComfyAPILatest(ComfyAPIBase):
	async def set_progress(value: float, total: float, preview: ImageInput, node_id: str):
		# ...

# comfy_api/v0_0_4.py
from comfy_api.latest import ComfyAPILatest
ComfyAPI = ComfyAPILatest

Version Lifecycle - Stable

When a version first moves to “stable”, we’ll create a new compatibility adapter that just links to the in development implementation. Each time we make a change to an existing API in the in development version, we’ll create a function in the latest adapter with the old signature to ensure it continues working.

# comfy_api/v0_0_3.py
from comfy_api.v0_0_4 import ComfyAPI as NextAPI
class ComfyAPI(NextAPI):
	async def set_progress(value: float, total: float, preview: ImageInput):
		await super().set_progress(
			value=value,
			total=total,
			preview=preview,
			node_id=await super().last_executed_node(),
		)

Async Calls

You may notice that the API calls use Python’s async semantics. This is necessary in order to begin supporting both parallel execution and out-of-process node execution, but it doesn’t actually change much about custom nodes.

Custom nodes that want to use the new API but don’t want to do anything fancy with regard to parallel execution just need to apply these two mechanical rules:

1. In function definitions, replace def with async def.

2. When calling an async function, add await before the function name.

There is no issue with calling non-async code from async call, so you can continue using whatever libraries you like. Calling async code from non-async code is a lot more difficult though. If you’re concerned about the move toward async calls, let us know!