Looks like the links got broken due to the blog moving to a different backend. I don't have access to edit the post currently, but I'll fix the workflow links when I can.
This is interesting. I worked on a flux region spatial control pipeline. I am trying to apply Lora to region-specific flux attention; however, the current system applies to the entire attention. I will study this in more detail. The regional solution I devised uses BBOX and Masking and feathering at the attention level, and we override the flux attention. Since this is now built into Comfy, I can try and adapt the custom pipeline to apply Lora to specific regions in Flux. Technically it should work using the technique showcased here.
For models like flux where you don't apply any weights to CLIP, you can skip Set CLIP Hooks entirely - instead, you can just connect the hooks on the Set Props (short for properties) nodes. The "apply-to-conds" option is simply there to simplify the process by not requiring to plug in the hooks on both CLIP and conditioning (on conditioning, the hooks serve as breadcrumbs for model weights during sampling).
The "Create Hook Model-As-LoRA" node is fairly transparent in its name that it will use a model the same way you can use a LoRA, implying that all the tools that would work for a normal LoRA Hook would work for the Model-As-LoRA hook. The goal was to make it clear it functions the same way as Create Hook LoRA node.
As for it being confusing and needing a tutorial, that is why this post exists - it IS the tutorial. There is no clear way to expose this functionality to the level of granularity it is now without some sort of compromise on confusion. There is also no built-in help menu at the moment for vanilla nodes (work in progress currently); I was hoping that would be implemented before this feature gets done, but it is not. Once it is, parts of this tutorial + more will be included in the built in help menu. These nodes definitely need a built-in help menu to not be cryptic, 100% agree on that.
The links to the workflows got fixed a couple hours ago - simply click on the .json files and it'll take you to a github page where you can download the .json. Substack (new backend for the blog) does not support uploading .json directly, and image metadata is stripped, so this was the only way for me to link.
If your brain run out of RAM while trying to assimilate both workflows here, check out [this post](https://www.andreszsogon.com/comfyui-lora-scheduling-with-hook-keyframes-nodes/) for a simplified Lora Scheduling workflow.
Great workflows btw, however, there's a performance issue with the Masking workflow that makes it 10x slower than usual.
PS. this comment plugin ignores formatting, how about switching to Disqus?
I cannot download the workflows, "page not found". Can you please share the link. Thank you.
Update: I have editing access now, and I've updated the links to .json workflows.
Looks like the links got broken due to the blog moving to a different backend. I don't have access to edit the post currently, but I'll fix the workflow links when I can.
Hi, same for me.
But it looks great, reading about it! And designing the workflow is good practice and it works.
This is interesting. I worked on a flux region spatial control pipeline. I am trying to apply Lora to region-specific flux attention; however, the current system applies to the entire attention. I will study this in more detail. The regional solution I devised uses BBOX and Masking and feathering at the attention level, and we override the flux attention. Since this is now built into Comfy, I can try and adapt the custom pipeline to apply Lora to specific regions in Flux. Technically it should work using the technique showcased here.
The new nodes are cool, but the naming... Woah, I cannot imagine any other naming scheme that is more confusing and missleading than this.
So when I hook a LoRA into Flux I use SetCLIPConditioning, although most Flux LoRAs do not even have CLIP lol
Averaging two model weights as a hook is called "lora" although model weight merging has nothing to do with LoRA.
The whole conditioning node set is confusing and requires a tutorial to understand it. Maybe just thinking for better names next time?
For models like flux where you don't apply any weights to CLIP, you can skip Set CLIP Hooks entirely - instead, you can just connect the hooks on the Set Props (short for properties) nodes. The "apply-to-conds" option is simply there to simplify the process by not requiring to plug in the hooks on both CLIP and conditioning (on conditioning, the hooks serve as breadcrumbs for model weights during sampling).
The "Create Hook Model-As-LoRA" node is fairly transparent in its name that it will use a model the same way you can use a LoRA, implying that all the tools that would work for a normal LoRA Hook would work for the Model-As-LoRA hook. The goal was to make it clear it functions the same way as Create Hook LoRA node.
As for it being confusing and needing a tutorial, that is why this post exists - it IS the tutorial. There is no clear way to expose this functionality to the level of granularity it is now without some sort of compromise on confusion. There is also no built-in help menu at the moment for vanilla nodes (work in progress currently); I was hoping that would be implemented before this feature gets done, but it is not. Once it is, parts of this tutorial + more will be included in the built in help menu. These nodes definitely need a built-in help menu to not be cryptic, 100% agree on that.
still not linked
The links to the workflows got fixed a couple hours ago - simply click on the .json files and it'll take you to a github page where you can download the .json. Substack (new backend for the blog) does not support uploading .json directly, and image metadata is stripped, so this was the only way for me to link.
🎥 Here is the workflow I made. Enjoy! 😊
https://openart.ai/workflows/kanana/flux-regional-masked-lora-and-prompt-magic-simple-workflow-for-8gb-vram/TT1jrDl5BzFpTeDFsLDK
got error using with flux. CLIPTextEncode
'NoneType' object has no attribute 'float'
I include the workflow and context in this ticket.
https://github.com/comfyanonymous/ComfyUI/issues/6082
Please post full error stack, this is not enough to debug.
Managed to make it work and share it here https://openart.ai/workflows/kanana/flux-regional-masked-lora-and-prompt-magic-simple-workflow-for-8gb-vram/TT1jrDl5BzFpTeDFsLDK
Thank you for your super fast reply. Please see the ticket here https://github.com/comfyanonymous/ComfyUI/issues/6082
hi,how should i get it?
You are a wizard! Excited to use this