Add documentation for Image Generation
- Refine StabilityAI classes - Add documentation
This commit is contained in:
Mark Pollack
@@ -24,6 +24,9 @@
|
||||
**** xref:api/clients/bedrock/bedrock-titan.adoc[]
|
||||
*** xref:api/clients/huggingface.adoc[]
|
||||
*** xref:api/clients/vertexai-chat.adoc[]
|
||||
** xref:api/imageclient.adoc[]
|
||||
*** xref:api/clients/image/openai-image.adoc[]
|
||||
*** xref:api/clients/image/stabilityai-image.adoc[]
|
||||
** xref:api/prompt.adoc[]
|
||||
** xref:api/output-parser.adoc[]
|
||||
** xref:api/vectordbs.adoc[]
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
= OpenAI Image Generation
|
||||
|
||||
|
||||
Spring AI supports ChatGPT's DALL-E, the Image generation model from OpenAI.
|
||||
|
||||
== Prerequisites
|
||||
|
||||
You will need to create an API key with OpenAI to access ChatGPT models.
|
||||
Create an account at https://platform.openai.com/signup[OpenAI signup page] and generate the token on the https://platform.openai.com/account/api-keys[API Keys page].
|
||||
The Spring AI project defines a configuration property named `spring.ai.openai.api-key` that you should set to the value of the `API Key` obtained from openai.com.
|
||||
Exporting an environment variable is one way to set that configuration property:
|
||||
|
||||
[source,shell]
|
||||
----
|
||||
export SPRING_AI_OPENAI_API_KEY=<INSERT KEY HERE>
|
||||
----
|
||||
|
||||
== Auto-configuration
|
||||
|
||||
Spring AI provides Spring Boot auto-configuration for the OpenAI Image Generation Client.
|
||||
To enable it add the following dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source, xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-openai-spring-boot-starter</artifactId>
|
||||
<version>0.8.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or to your Gradle `build.gradle` build file.
|
||||
|
||||
[source,groovy]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-openai-spring-boot-starter:0.8.0-SNAPSHOT'
|
||||
}
|
||||
----
|
||||
|
||||
TIP: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
|
||||
=== Image Generation Properties
|
||||
|
||||
The prefix `spring.ai.openai` is used as the property prefix that lets you connect to OpenAI.
|
||||
|
||||
[cols="3,5,1"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
| spring.ai.openai.base-url | The URL to connect to | https://api.openai.com
|
||||
| spring.ai.openai.api-key | The API Key | -
|
||||
|====
|
||||
|
||||
The prefix `spring.ai.openai.image` is the property prefix that lets you configure the `ImageClient` implementation for OpenAI.
|
||||
|
||||
[cols="3,5,1"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
| spring.ai.openai.chat.base-url | Optional overrides the spring.ai.openai.base-url to provide chat specific url | -
|
||||
| spring.ai.openai.chat.api-key | Optional overrides the spring.ai.openai.api-key to provide chat specific api-key | -
|
||||
| spring.ai.openai.image.options.n | The number of images to generate. Must be between 1 and 10. For dall-e-3, only n=1 is supported. | -
|
||||
| spring.ai.openai.image.options.model | The model to use for image generation. | OpenAiImageApi.DEFAULT_IMAGE_MODEL
|
||||
| spring.ai.openai.image.options.quality | The quality of the image that will be generated. HD creates images with finer details and greater consistency across the image. This parameter is only supported for dall-e-3. | -
|
||||
| spring.ai.openai.image.options.response_format | The format in which the generated images are returned. Must be one of URL or b64_json. | -
|
||||
| `spring.ai.openai.image.options.size` | The size of the generated images. Must be one of 256x256, 512x512, or 1024x1024 for dall-e-2. Must be one of 1024x1024, 1792x1024, or 1024x1792 for dall-e-3 models. | -
|
||||
| `spring.ai.openai.image.options.size_width` | The width of the generated images. Must be one of 256, 512, or 1024 for dall-e-2. | -
|
||||
| `spring.ai.openai.image.options.size_height`| The height of the generated images. Must be one of 256, 512, or 1024 for dall-e-2. | -
|
||||
| `spring.ai.openai.image.options.style` | The style of the generated images. Must be one of vivid or natural. Vivid causes the model to lean towards generating hyper-real and dramatic images. Natural causes the model to produce more natural, less hyper-real looking images. This parameter is only supported for dall-e-3. | -
|
||||
| `spring.ai.openai.image.options.user` | A unique identifier representing your end-user, which can help OpenAI to monitor and detect abuse. | -
|
||||
|====
|
||||
|
||||
=== Image Options [[image-options]]
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-openai/src/main/java/org/springframework/ai/openai/OpenAiImageOptions.java[OpenAiImageOptions.java] provides model configurations, such as the model to use, the quality, the size, etc.
|
||||
|
||||
On start-up, the default options can be configured with the `OpenAiImageClient(OpenAiImageApi openAiImageApi)` constructor and the `withDefaultOptions(OpenAiImageOptions defaultOptions)` method. Alternatively, use the `spring.ai.openai.image.options.*` properties described previously.
|
||||
|
||||
At run-time you can override the default options by adding new, request specific, options to the `ImagePrompt` call.
|
||||
For example to override the OpenAI specific options such as quality and the number of images to create, use the following code example:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ImageResponse response = openaiImageClient.call(
|
||||
new ImagePrompt("A light cream colored mini golden doodle",
|
||||
OpenAiImageOptions.builder()
|
||||
.withQuality("hd")
|
||||
.withN(4)
|
||||
.withHeight(1024)
|
||||
.withWidth(1024).build())
|
||||
|
||||
);
|
||||
----
|
||||
|
||||
TIP: In addition to the model specific https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-openai/src/main/java/org/springframework/ai/openai/OpenAiImageOptions.java[OpenAiImageOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/image/ImageOptions.java[ImageOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/image/ImageOptionsBuilder.java[ImageOptionsBuilder#builder()].
|
||||
@@ -0,0 +1,99 @@
|
||||
= Stability AI Image Generation
|
||||
|
||||
Spring AI supports Stability AI's https://platform.stability.ai/docs/api-reference#tag/v1generation[text to image generation model].
|
||||
|
||||
== Prerequisites
|
||||
|
||||
You will need to create an API key with Stability AI to access their AI models, follow their https://platform.stability.ai/docs/getting-started/authentication[Getting Started documentation].
|
||||
|
||||
The Spring AI project defines a configuration property named `spring.ai.stabilityai.api-key` that you should set to the value of the `API Key` obtained from Stability AI.
|
||||
Exporting an environment variable in one way to set that configuration property.
|
||||
|
||||
[source,shell]
|
||||
----
|
||||
export SPRING_AI_STABILITYAI_API_KEY=<INSERT KEY HERE>
|
||||
----
|
||||
|
||||
== Auto-configuration
|
||||
|
||||
Spring AI provides Spring Boot auto-configuration for the Stability AI Image Generation Client.
|
||||
To enable it add the following dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source, xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-stability-ai-spring-boot-starter</artifactId>
|
||||
<version>0.8.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or to your Gradle `build.gradle` build file.
|
||||
|
||||
[source,groovy]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-stability-ai-spring-boot-starter:0.8.0-SNAPSHOT'
|
||||
}
|
||||
----
|
||||
|
||||
TIP: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
|
||||
=== Image Generation Properties
|
||||
|
||||
The prefix `spring.ai.stabilityai` is used as the property prefix that lets you connect to Stability AI.
|
||||
|
||||
[cols="3,5,1"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
|
||||
| spring.ai.stabilityai.base-url | The URL to connect to | https://api.stability.ai/v1
|
||||
| spring.ai.stabilityai.api-key | The API Key | -
|
||||
|====
|
||||
|
||||
The prefix `spring.ai.stabilityai.image` is the property prefix that lets you configure the `ImageClient` implementation for Stability AI.
|
||||
|
||||
[cols="2,5,1"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
|
||||
| `spring.ai.stabilityai.image.base-url` | Optional overrides the spring.ai.openai.base-url to provide a specific url | `https://api.stability.ai/v1`
|
||||
| `spring.ai.stabilityai.image.api-key` | Optional overrides the spring.ai.openai.api-key to provide a specific api-key | -
|
||||
| `spring.ai.stabilityai.image.option.n` | The number of images to be generated. Must be between 1 and 10. | 1
|
||||
| `spring.ai.stabilityai.image.option.model` | The engine/model to use in Stability AI. The model is passed in the URL as a path parameter. | `stable-diffusion-v1-6`
|
||||
| `spring.ai.stabilityai.image.option.width` | Width of the image to generate, in pixels, in an increment divisible by 64. Engine-specific dimension validation applies. | 512
|
||||
| `spring.ai.stabilityai.image.option.height` | Height of the image to generate, in pixels, in an increment divisible by 64. Engine-specific dimension validation applies.| 512
|
||||
| `spring.ai.stabilityai.image.option.responseFormat` | The format in which the generated images are returned. Must be "application/json" or "image/png". | -
|
||||
| `spring.ai.stabilityai.image.option.cfg_scale` | The strictness level of the diffusion process adherence to the prompt text. Range: 0 to 35. | 7
|
||||
| `spring.ai.stabilityai.image.option.clip_guidance_preset` | Pass in a style preset to guide the image model towards a particular style. This list of style presets is subject to change. | `NONE`
|
||||
| `spring.ai.stabilityai.image.option.sampler` | Which sampler to use for the diffusion process. If this value is omitted, an appropriate sampler will be automatically selected. | -
|
||||
| `spring.ai.stabilityai.image.option.seed` | Random noise seed (omit this option or use 0 for a random seed). Valid range: 0 to 4294967295. | 0
|
||||
| `spring.ai.stabilityai.image.option.steps` | Number of diffusion steps to run. Valid range: 10 to 50. | 30
|
||||
| `spring.ai.stabilityai.image.option.style_preset` | Pass in a style preset to guide the image model towards a particular style. This list of style presets is subject to change. | -
|
||||
|====
|
||||
|
||||
|
||||
=== Image Options [[image-options]]
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-stabilityai/src/main/java/org/springframework/ai/stabilityai/api/StabilityAiImageOptions.java[StabilityAiImageOptions.java] provides model configurations, such as the model to use, the style, the size, etc.
|
||||
|
||||
On start-up, the default options can be configured with the `StabilityAiImageClient(StabilityAiApi stabilityAiApi, StabilityAiImageOptions options)` constructor. Alternatively, use the `spring.ai.openai.image.options.*` properties described previously.
|
||||
|
||||
At run-time you can override the default options by adding new, request specific, options to the `ImagePrompt` call.
|
||||
For example to override the Stability AI specific options such as quality and the number of images to create, use the following code example:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ImageResponse response = openaiImageClient.call(
|
||||
new ImagePrompt("A light cream colored mini golden doodle",
|
||||
StabilityAiImageOptions.builder()
|
||||
.withStylePreset("cinematic")
|
||||
.withN(4)
|
||||
.withHeight(1024)
|
||||
.withWidth(1024).build())
|
||||
|
||||
);
|
||||
----
|
||||
|
||||
TIP: In addition to the model specific https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-stabilityai/src/main/java/org/springframework/ai/stabilityai/api/StabilityAiImageOptions.java[StabilityAiImageOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/image/ImageOptions.java[ImageOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/image/ImageOptionsBuilder.java[ImageOptionsBuilder#builder()].
|
||||
@@ -0,0 +1,171 @@
|
||||
[[ImageClient]]
|
||||
= Image Generation API
|
||||
|
||||
|
||||
The `Spring Image Generation API` is designed to be a simple and portable interface for interacting with various xref:concepts.adoc#_models[AI Models] specialized in image generation, allowing developers to switch between different image-related models with minimal code changes.
|
||||
This design aligns with Spring's philosophy of modularity and interchangeability, ensuring developers can quickly adapt their applications to different AI capabilities related to image processing.
|
||||
|
||||
Additionally, with the support of companion classes like `ImagePrompt` for input encapsulation and `ImageResponse` for output handling, the Image Generation API unifies the communication with AI Models dedicated to image generation.
|
||||
It manages the complexity of request preparation and response parsing, offering a direct and simplified API interaction for image-generation functionalities.
|
||||
|
||||
The Spring Image Generation API is built on top of the Spring AI `Generic Model API`, providing image-specific abstractions and implementations.
|
||||
|
||||
== API Overview
|
||||
|
||||
This section provides a guide to the Spring Image Generation API interface and associated classes.
|
||||
|
||||
== Image Client
|
||||
|
||||
Here is the link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/image/ImageClient.java[ImageClient] interface definition:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@FunctionalInterface
|
||||
public interface ImageClient extends ModelClient<ImagePrompt, ImageResponse> {
|
||||
|
||||
ImageResponse call(ImagePrompt request);
|
||||
|
||||
}
|
||||
----
|
||||
|
||||
=== ImagePrompt
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/image/ImagePrompt.java[ImagePrompt] is a `ModelRequest` that encapsulates a list of https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/image/ImageMessage.java[ImageMessage] objects and optional model request options.
|
||||
The following listing shows a truncated version of the `ImagePrompt` class, excluding constructors and other utility methods:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
public class ImagePrompt implements ModelRequest<List<ImageMessage>> {
|
||||
|
||||
private final List<ImageMessage> messages;
|
||||
|
||||
private ImageOptions imageModelOptions;
|
||||
|
||||
@Override
|
||||
public List<ImageMessage> getInstructions() {...}
|
||||
|
||||
@Override
|
||||
public ImageOptions getOptions() {...}
|
||||
|
||||
// constructors and utility methods omitted
|
||||
}
|
||||
----
|
||||
|
||||
==== ImageMessage
|
||||
|
||||
The `ImageMessage` class encapsulates the text to use and the weight that the text should have in influencing the generated image. For models that support weights, they can be positive or negative.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
public class ImageMessage {
|
||||
|
||||
private String text;
|
||||
|
||||
private Float weight;
|
||||
|
||||
public String getText() {...}
|
||||
|
||||
public Float getWeight() {...}
|
||||
|
||||
// constructors and utility methods omitted
|
||||
|
||||
----
|
||||
|
||||
==== ImageOptions
|
||||
|
||||
Represents the options that can be passed to the Image generation model. The `ImageOptions` class extends the `ModelOptions` interface and is used to define few portable options that can be passed to the AI model.
|
||||
|
||||
The `ImageOptions` class is defined as follows:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
public interface ImageOptions extends ModelOptions {
|
||||
|
||||
Integer getN();
|
||||
|
||||
String getModel();
|
||||
|
||||
Integer getWidth();
|
||||
|
||||
Integer getHeight();
|
||||
|
||||
String getResponseFormat(); // openai - url or base64 : stability ai byte[] or base64
|
||||
|
||||
}
|
||||
----
|
||||
|
||||
Additionally, every model specific ImageClient implementation can have its own options that can be passed to the AI model. For example, the OpenAI Image Generation model has its own options like `quality`, `style`, etc.
|
||||
|
||||
|
||||
This is a powerful feature that allows developers to use model specific options when starting the application and then override them with at runtime using the `ImagePrompt`.
|
||||
|
||||
|
||||
=== ImageResponse
|
||||
|
||||
The structure of the `ChatResponse` class is as follows:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
public class ImageResponse implements ModelResponse<ImageGeneration> {
|
||||
|
||||
private final ImageResponseMetadata imageResponseMetadata;
|
||||
|
||||
private final List<ImageGeneration> imageGenerations;
|
||||
|
||||
@Override
|
||||
public ImageGeneration getResult() {
|
||||
// get the first result
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<ImageGeneration> getResults() {...}
|
||||
|
||||
@Override
|
||||
public ImageResponseMetadata getMetadata() {...}
|
||||
|
||||
// other methods omitted
|
||||
|
||||
}
|
||||
----
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/image/ImageResponse.java[ImageResponse] class holds the AI Model's output, with each `ImageGeneration` instance containing one of potentially multiple outputs resulting from a single prompt.
|
||||
|
||||
The `ImageResponse` class also carries a `ImageResponseMetadata` metadata about the AI Model's response.
|
||||
|
||||
=== ImageGeneration
|
||||
|
||||
Finally, the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/image/ImageGeneration.java[ImageGeneration] class extends from the `ModelResult` to represent the output response and related metadata about this result:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
public class ImageGeneration implements ModelResult<Image> {
|
||||
|
||||
private ImageGenerationMetadata imageGenerationMetadata;
|
||||
|
||||
private Image image;
|
||||
|
||||
@Override
|
||||
public Image getOutput() {...}
|
||||
|
||||
@Override
|
||||
public ImageGenerationMetadata getMetadata() {...}
|
||||
|
||||
// other methods omitted
|
||||
|
||||
}
|
||||
----
|
||||
|
||||
== Available Implementations
|
||||
|
||||
`ImageClient` implementations are provided for the following Model providers:
|
||||
|
||||
* xref:api/clients/image/openai-image.adoc[OpenAI Image Generation]
|
||||
* xref:api/clients/image/stabilityai-image.adoc[StabilityAI Image Generation]
|
||||
|
||||
== API Docs
|
||||
|
||||
You can find the Javadoc https://docs.spring.io/spring-ai/docs/current-SNAPSHOT/[here].
|
||||
|
||||
== Feedback and Contributions
|
||||
|
||||
The project's https://github.com/spring-projects/spring-ai/discussions[GitHub discussions] is a great place to send feedback.
|
||||
@@ -3,6 +3,7 @@
|
||||
|
||||
This section offers jumping off points for how to get started using Spring AI.
|
||||
|
||||
[#_dependency_management]
|
||||
== Dependency Management
|
||||
|
||||
The Spring AI project provides artifacts in the Spring Milestone and Snapshot repositories.
|
||||
|
||||
Reference in New Issue
Block a user