ComfyUI Workflow into API with an Actual Example in Next.js

ComfyUI is an advanced graphical user interface for building intricate image generation workflows, particularly in AI and machine learning. While it excels in visual experimentation, integrating these workflows into applications requires an API-based approach. This tutorial provides a step-by-step guide to converting your ComfyUI workflow into a functional API using Next.js, complete with actual code examples.

Prerequisites
Understanding the Workflow
Setting Up the Next.js Application
Creating API Routes
Establishing WebSocket Connections
Modifying the Workflow Dynamically
Tracking Progress
Retrieving and Serving Images
Putting It All Together
Conclusion

Prerequisites

Before we begin, ensure you have the following:

Node.js (v12 or higher) and npm installed.
Next.js project set up. If not, we'll cover how to create one.
ComfyUI installed and running on your machine.
Basic understanding of JavaScript/TypeScript and Next.js.
Familiarity with RESTful APIs and WebSockets.

Understanding the Workflow

A ComfyUI workflow consists of interconnected nodes that define the image generation process. Each node performs a specific function, such as applying a filter or generating noise. By converting this workflow into an API, you enable external applications to interact with it programmatically.

Setting Up the Next.js Application

Initialize a New Next.js Project

First, create a new Next.js project. In your terminal, run:

npx create-next-app comfyui-api-tutorial

Navigate to the project directory:

cd comfyui-api-tutorial

Install Required Dependencies

We'll need additional packages for WebSocket communication and handling HTTP requests:

npm install ws uuid

ws: A WebSocket library for Node.js to handle WebSocket connections.
uuid: For generating unique client IDs.

Creating API Routes

Next.js provides a convenient way to create API routes that run on the server side.

API Route to Queue a Prompt

Create a new file at pages/api/queuePrompt.js:

// pages/api/queuePrompt.js
 
import { v4 as uuidv4 } from 'uuid';
 
export default async function handler(req, res) {
  if (req.method === 'POST') {
    const serverAddress = '127.0.0.1:8188';
    const { prompt } = req.body;
    const clientId = uuidv4();
 
    const payload = {
      prompt,
      client_id: clientId,
    };
 
    try {
      const response = await fetch(`http://${serverAddress}/prompt`, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(payload),
      });
 
      if (!response.ok) {
        throw new Error(`Error queueing prompt: ${response.statusText}`);
      }
 
      const data = await response.json();
 
      res.status(200).json({ promptId: data.prompt_id, clientId });
    } catch (error) {
      console.error('Error:', error);
      res.status(500).json({ error: error.message });
    }
  } else {
    res.setHeader('Allow', 'POST');
    res.status(405).end('Method Not Allowed');
  }
}

API Route to Retrieve History

Create a new file at pages/api/history/[promptId].js:

// pages/api/history/[promptId].js
 
export default async function handler(req, res) {
  const { promptId } = req.query;
  const serverAddress = '127.0.0.1:8188';
 
  try {
    const response = await fetch(`http://${serverAddress}/history/${promptId}`);
 
    if (!response.ok) {
      throw new Error(`Error retrieving history: ${response.statusText}`);
    }
 
    const data = await response.json();
 
    res.status(200).json(data);
  } catch (error) {
    console.error('Error:', error);
    res.status(500).json({ error: error.message });
  }
}

Establishing WebSocket Connections

To receive real-time updates from ComfyUI, we need to establish a WebSocket connection.

WebSocket Utility

Create a new directory utils and a file websocket.js:

// utils/websocket.js
 
import WebSocket from 'ws';
import { v4 as uuidv4 } from 'uuid';
 
export function openWebSocketConnection() {
  const serverAddress = '127.0.0.1:8188';
  const clientId = uuidv4();
 
  const ws = new WebSocket(`ws://${serverAddress}/ws?clientId=${clientId}`);
 
  ws.on('open', () => {
    console.log('WebSocket connection established.');
  });
 
  ws.on('close', () => {
    console.log('WebSocket connection closed.');
  });
 
  ws.on('error', (error) => {
    console.error('WebSocket error:', error);
  });
 
  return { ws, clientId };
}

Modifying the Workflow Dynamically

We need to adjust the workflow based on the user's input.

Workflow Utility

Create a new file utils/workflow.js:

// utils/workflow.js
 
import crypto from 'crypto';
 
export function modifyWorkflow(workflowJson, positivePrompt, negativePrompt = '') {
  const workflow = JSON.parse(workflowJson);
 
  // Map node IDs to class types
  const idToClassType = {};
  Object.entries(workflow).forEach(([id, node]) => {
    idToClassType[id] = node.class_type;
  });
 
  // Find the KSampler node
  const kSamplerNodeId = Object.keys(idToClassType).find(
    (id) => idToClassType[id] === 'KSampler'
  );
 
  if (kSamplerNodeId) {
    const kSamplerNode = workflow[kSamplerNodeId];
 
    // Set a random seed
    kSamplerNode.inputs.seed = crypto.randomInt(1e14, 1e15 - 1);
 
    // Update positive prompt
    const positiveInputId = kSamplerNode.inputs.positive[0];
    workflow[positiveInputId].inputs.text = positivePrompt;
 
    // Update negative prompt if provided
    if (negativePrompt) {
      const negativeInputId = kSamplerNode.inputs.negative[0];
      workflow[negativeInputId].inputs.text = negativePrompt;
    }
  }
 
  return JSON.stringify(workflow);
}

Tracking Progress

We can listen to WebSocket messages to track the progress of the image generation.

Progress Utility

Update utils/websocket.js:

// utils/websocket.js
 
// ... existing code ...
 
export function trackProgress(ws, promptId, onProgressCallback, onCompleteCallback) {
  ws.on('message', (data) => {
    const message = JSON.parse(data);
 
    if (message.type === 'progress' && message.data.prompt_id === promptId) {
      const progress = message.data;
      onProgressCallback(progress);
    }
 
    if (
      (message.type === 'execution_complete' ||
        message.type === 'processing_complete') &&
      message.data.prompt_id === promptId
    ) {
      ws.close();
      onCompleteCallback();
    }
  });
}

Retrieving and Serving Images

After the image generation is complete, retrieve the images from ComfyUI.

Image Retrieval Utility

Create a new file utils/images.js:

// utils/images.js
 
export async function getGeneratedImages(serverAddress, promptId) {
  const response = await fetch(`http://${serverAddress}/history/${promptId}`);
 
  if (!response.ok) {
    throw new Error(`Error retrieving history: ${response.statusText}`);
  }
 
  const history = await response.json();
 
  const promptHistory = history[promptId];
  const images = [];
 
  for (const nodeOutput of Object.values(promptHistory.outputs)) {
    if (nodeOutput.images) {
      images.push(...nodeOutput.images);
    }
  }
 
  return images;
}

API Route to Serve Images

Create a new file at pages/api/images/[promptId].js:

// pages/api/images/[promptId].js
 
import { getGeneratedImages } from '../../../utils/images';
 
export default async function handler(req, res) {
  const { promptId } = req.query;
  const serverAddress = '127.0.0.1:8188';
 
  try {
    const images = await getGeneratedImages(serverAddress, promptId);
 
    res.status(200).json({ images });
  } catch (error) {
    console.error('Error:', error);
    res.status(500).json({ error: error.message });
  }
}

Putting It All Together

Now, let's integrate all the pieces in a single API route that handles the entire flow.

Complete API Route

Create a new file pages/api/generateImage.js:

// pages/api/generateImage.js
 
import { openWebSocketConnection, trackProgress } from '../../utils/websocket';
import { modifyWorkflow } from '../../utils/workflow';
import { getGeneratedImages } from '../../utils/images';
 
export default async function handler(req, res) {
  if (req.method === 'POST') {
    const serverAddress = '127.0.0.1:8188';
    const { positivePrompt, negativePrompt } = req.body;
 
    try {
      const { ws, clientId } = openWebSocketConnection();
 
      // Load your workflow JSON string
      const workflowJson = getYourWorkflowJson();
 
      // Modify the workflow with user prompts
      const modifiedWorkflow = modifyWorkflow(
        workflowJson,
        positivePrompt,
        negativePrompt
      );
 
      // Queue the modified prompt
      const promptResponse = await fetch(`http://${serverAddress}/prompt`, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ prompt: modifiedWorkflow, client_id: clientId }),
      });
 
      if (!promptResponse.ok) {
        throw new Error(`Error queueing prompt: ${promptResponse.statusText}`);
      }
 
      const promptData = await promptResponse.json();
      const promptId = promptData.prompt_id;
 
      // Track progress
      trackProgress(
        ws,
        promptId,
        (progress) => {
          console.log(`Progress: ${progress.value} / ${progress.max}`);
        },
        async () => {
          // Image generation completed
          const images = await getGeneratedImages(serverAddress, promptId);
          res.status(200).json({ images });
        }
      );
    } catch (error) {
      console.error('Error:', error);
      res.status(500).json({ error: error.message });
    }
  } else {
    res.setHeader('Allow', 'POST');
    res.status(405).end('Method Not Allowed');
  }
}
 
// Dummy function to simulate retrieving your workflow JSON
function getYourWorkflowJson() {
  // Replace this with actual code to retrieve your workflow
  // For example, read from a file or database
  return JSON.stringify({
    // ... your ComfyUI workflow JSON ...
  });
}

Client-Side Usage

You can create a simple page to interact with the API.

// pages/index.js
 
import { useState } from 'react';
 
export default function Home() {
  const [positivePrompt, setPositivePrompt] = useState('');
  const [negativePrompt, setNegativePrompt] = useState('');
  const [images, setImages] = useState([]);
 
  const generateImage = async () => {
    const response = await fetch('/api/generateImage', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ positivePrompt, negativePrompt }),
    });
 
    if (response.ok) {
      const data = await response.json();
      setImages(data.images);
    } else {
      console.error('Error generating image:', response.statusText);
    }
  };
 
  return (
    <div>
      <h1>ComfyUI Image Generator</h1>
      <input
        type="text"
        placeholder="Positive Prompt"
        value={positivePrompt}
        onChange={(e) => setPositivePrompt(e.target.value)}
      />
      <input
        type="text"
        placeholder="Negative Prompt"
        value={negativePrompt}
        onChange={(e) => setNegativePrompt(e.target.value)}
      />
      <button onClick={generateImage}>Generate Image</button>
      <div>
        {images.map((image, index) => (
          <img
            key={index}
            src={`data:image/png;base64,${image.base64}`}
            alt={`Generated ${index}`}
          />
        ))}
      </div>
    </div>
  );
}

Or, You Can Use ComfyUIAsAPI.com's Service

If you don't want to deal with the hassle of setting up the API, you can use ComfyUIAsAPI.com's service.

All you need is to upload your ComfyUI workflow .json file and get a ready-to-use API. We also support SDKs for all the popular languages. You just need a few lines of code to integrate it into your project.

Conclusion

By following this step-by-step tutorial, you've transformed your ComfyUI workflow into a functional API using Next.js. This integration allows for seamless interaction between your image generation workflow and other applications, enabling dynamic input and real-time feedback. Leveraging Next.js API routes and server-side capabilities provides a robust platform for building interactive and responsive AI-powered applications.