Revolutionize Your Image Processing with a Free, High-Performance Diffusion App Powered by GPU Technology

The Stable Diffusion model is an incredibly powerful tool that can generate high-quality images from a single sentence. However, the model requires the use of GPUs to produce results within a reasonable timeframe. Unfortunately, GPUs can be costly and in short supply, making it difficult for many users to utilize Stable Diffusion to its full potential. As a result, many users resort to paying for cloud-based web applications like Dream Studio to access the model.

Fortunately, there is a way to leverage Google Collab’s free GPUs and build a simple web app that runs Stable Diffusion without any cost. With this app, users can type in a prompt, and within 10 seconds, the generated image will be displayed on their browser. In this guide, we'll walk you through the steps to create your own web app, providing a cost-free solution for using Stable Diffusion in the cloud.

If you lack coding experience or simply wish to use the application, feel free to proceed directly to the "Use the Application" section. However, for those who would like to learn more about the inner workings of the app, let's dive in and explore the details.

To build the Stable Diffusion web app, we will use Flask, Jinja, and ngrok. These tools will enable us to generate and display images for user-submitted prompts in a seamless and user-friendly manner.

Flask is a Python-based microweb framework that allows us to run different Python functions based on different routes. When a user submits a prompt via the Stable Diffusion web interface, our Flask app will run a Python function that generates an image based on that prompt. The resulting image will then be inserted into an HTML document using Jinja, a templating engine that simplifies the process of generating and returning HTML documents. The Flask application will be hosted on Google Colab's servers, and to make it accessible to users, we will use ngrok, a reverse proxy that allows us to expose our application to the internet by providing a public URL. This approach will make our Stable Diffusion web app accessible to users from anywhere with an internet connection.

In summary, our schematic for building the Stable Diffusion web app will utilize Flask and Jinja for generating and returning images, respectively, and ngrok for making the application accessible to users via a public URL.

The process of generating an image using Ngrok and Colab's servers involves the following steps:

1. The user submits a caption to Ngrok, which acts as an intermediary.
2. Ngrok forwards the request to Colab's servers.
3. The server uses Flask, a micro web framework for Python, to execute the generate_image() function.
4. The generate_image() function runs the image generation process on a Colab GPU, resulting in a high-performance output.
5. The generated image is inserted into an HTML template using Jinja, a web template engine for Python.
6. The resulting HTML document, displaying the generated image, is returned to the user in as little as 10 seconds.

With a clear understanding of the overall process, we can now take a closer look at the code in the associated Colab notebook and witness the steps in action.

Step-by-Step Guide: Building Your Own Stable Diffusion Web App with Ease

To build the Stable Diffusion web app, you need to follow these steps:

Step 1: Install Dependencies

Before we get started, we must install all the dependencies required to run the Stable Diffusion model. Here's how to do it:

• Install the diffusers library, which is necessary for running the Stable Diffusion model, as well as transformers, scipy, ftfy, and accelerate packages, by executing the following command in your Colab notebook:

pip install diffusers==0.10.2 transformers scipy ftfy accelerate

• Install flask_ngrok, which is necessary for running a Flask application using ngrok, by executing the following command in your Colab notebook:

pip install flask_ngrok

Step 2: Create an ngrok Account

To use ngrok's services and run a Flask application, you must create an ngrok account and obtain an authentication token. Here's how:

• Go to ngrok.com, click "Sign up for free," and create an account.
• Verify your email to activate your account.
• Go to the ngrok dashboard and copy your Authtoken to authenticate your account for future use.

To register your ngrok account in the configuration, follow these steps:

1. Open the Colab notebook and locate the cell where the ngrok authentication token is required.
2. Replace "YOUR-TOKEN-HERE" with your ngrok authentication token.
3. Run the cell to register your account in ngrok's configuration.

To replace the "YOUR-TOKEN-HERE" placeholder, paste your ngrok authentication token in the cell where the following command is present:

ngrok authtoken YOUR-AUTHTOKEN-HERE

Make sure to run the cell after pasting your authentication token to complete the registration process.

Step 3: Create the App Files

After installing the dependencies and creating an ngrok account, the next step is to create two essential files that the Flask application will use: a CSS file for styling and an HTML file that Flask will return in the HTTP response.

The CSS file is straightforward and provides simple styling for the web application. The HTML file, on the other hand, has a unique structure and may appear nonstandard to some. If you examine the HTML file, you'll see an image source that looks like this:

src="{{ generated_image|default("https://images.squarespace-cdn.com/abc.png?format=512w", true) }}"

The Jinja templating system allows us to replace the image displayed on our Stable Diffusion web app dynamically. By providing a default value when no image has been generated yet, we can avoid creating and returning an entirely new HTML file for every new caption submission. Instead, we need to save the generated image, and Jinja will take care of the rest. By doing this, the web app will display the generated image when it's available, or the default image when it's not.

To create the necessary CSS and HTML files for this feature, simply run the cells in the Step 2 section of the Colab notebook.

Step 4: Build and Execute the Flask App

In this step, we will be constructing and launching the Flask application. To do so, we will need to import several essential modules at the beginning of the code cell. These modules will enable us to serve the application, generate images, and convert image types as needed.

from flask_ngrok import run_with_ngrok
from flask import Flask, render_template, send_file, request
import torch
from diffusers import StableDiffusionPipeline
import base64
from io import BytesIO

In the next step, we initialize the Stable Diffusion (1.5) model with the diffusers library and transfer it to the GPU. This action loads the model into the system's memory, allowing it to process incoming requests efficiently.

pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", revision="fp16", torch_dtype=torch.float16)

pipe.to("cuda")

Next, we create and start up a Flask object and set it to run with ngrok.

app = Flask(__name__, template_folder='.')

run_with_ngrok(app)

Creating the Initial Render Endpoint

Defining the application is an essential step in web development. In order to achieve this, we can utilize the app.route() decorator to associate different endpoints with corresponding Python functions. These functions will execute when a request is made to the specified endpoint.

To begin with, let's create a basic function to return the initial web page that appears when the app is accessed. We will map the route / to this function which will then return the index.html file we generated previously.

@app.route('/')

def initial():

  return render_template('index.html')

In Flask, the render_template() function is used to inform the server that a Jinja template is being returned, which needs to be populated dynamically. If no other arguments are provided to the function, it will use the defaults specified in the index.html file. This ensures that the template is displayed correctly without any errors.

The subsequent step involves creating an endpoint for caption submission. Subsequently, the function responsible for handling every instance of caption submission is defined.

@app.route('/submit-caption', methods=['POST'])

def generate_image():

  prompt = request.form['caption-input']

  print(f"Generating an image of {prompt}")

  image = pipe(prompt).images[0]

  print("Image generated! Converting image ...")

  buffered = BytesIO()

  image.save(buffered, format="PNG")

  img_str = base64.b64encode(buffered.getvalue())

  b = "data:image/png;base64," + str(img_str)[2:-1]

  print("Sending image ...")

  return render_template('index.html', generated_image=b)

To direct requests for the path /submit-caption to the appropriate function, we use app.route(). By specifying the allowable HTTP methods as POST instead of the default GET, we allow users to submit captions for processing.

Moving on, we define the prompt variable. In index.html, the <form> element contains an <input> tag with the name property set as caption-input. We use request.form['caption-input'] to capture the text entered in this textbox and assign it to the prompt variable.

Subsequently, we pass the prompt through the Stable Diffusion model, which we had already loaded at the beginning of the file, using image = pipe(prompt).images[0]. The resulting image is then assigned to the image variable. It's worth noting that we loaded the Stable Diffusion model at the top of the file instead of within the function to avoid reloading it for every request.

After obtaining the image, we proceed to convert it into a base-64 string. This conversion enables us to conveniently embed the image directly into the HTML file, which is then sent in the response. To indicate that the <img> tag in the HTML file contains a raw image rather than a URL, we append "data:image/png;base64," to the string.

The next step involves populating the index.html template with the generated image string. Finally, we send the populated template in the HTTP response.

return render_template('index.html', generated_image=b)

We use the render_template() function in order to return the index.html file we created above. This is similar to the previous endpoint we defined above, except this time we are actually using Jinja's templating by dynamically replacing generated_image in the template with the image string b.

Run the app

Last but certainly not least, we use app.run() to actually run the Flask application. After executing app.run(), the Flask app will start and a localhost URL at which it can be locally accessed (on the server) will be printed in the cell output. After this, an ngrok URL at which the app can be publicly accessed will be printed.

Use the Stable Diffusion web app

If you jumped down from the beginning of the article / did not follow along with the above section, go to the Setup subsection below. Otherwise, jump down to the Use the app subsection.

Setup

1. Go to the Google Colab notebook.
2. Click Runtime > Change runtime type in the toolbar at the top of the screen and make sure that GPU is listed under Hardware accelerator, selecting it if not.
3. Complete Step 2 in the Colab notebook (no need to run the authtoken cell).
4. Click Runtime > Run all in the toolbar at the top of the screen
5. Jump down to Step 5 in the Colab notebook, and then move on to the next subsection.

Use the app

To go to the Stable Diffusion application, simply click the ngrok URL that is output from the last cell in Step 4.

You will receive a notification that you are visiting a website served via ngrok. Click "Visit Site" to advance to the application.

To use the app, simply enter a prompt in the textbox and click "Create". After about 10 seconds, you will see the generated image appear. Below we can see the results the caption "an image of a beautiful sunny landscape, grassy field with ravine and mountain, Greg Rutkowski, romanticism, high detail, painting, digital art, trending on artstation"

Make sure not to submit a second prompt/caption until you see the image from previous one appears in your browser!