Text and Images Together, Multimodal Output

Every chapter in this module so far has asked for one kind of output at a time: text, or an image. This chapter is about asking for both in the same response, genuinely mixed, not two separate calls stitched together afterwards.

A different shape of request

Everything before this chapter called a specific generation method, generate_text(), generate_image(), matched to a specific kind of output. Multimodal output doesn’t work that way, since the response itself is a mix; there’s a new, more general method for it:

use WordPress\AiClient\Messages\Enums\ModalityEnum;

$result = wp_ai_client_prompt( 'Create a recipe for a chocolate cake and include photos for the steps.' )
    ->as_output_modalities( ModalityEnum::text(), ModalityEnum::image() )
    ->generate_result();

ModalityEnum::text() and ModalityEnum::image() are the two modalities this book actually uses, but they’re not the only two that exist. Audio and video are modalities too, in principle as_output_modalities() could mix those in the same way. This book doesn’t cover them here for the same reason Chapter 17 holds off on speech and video generation entirely, no provider currently supports them, so there’s nothing to test. Module 10 revisits this exact method once that changes, rather than treating it as new all over again.

as_output_modalities() tells the model which kinds of content you’re willing to receive, text, image, or both together. generate_result() is new too, it’s not tied to one modality the way generate_text() or generate_image() are, since the response might contain either, or a mix of both in sequence.

Reading a mixed response

The result isn’t a plain string or a single File object this time, it’s a message made up of parts, and each part could be text or a file. Each part has a getType() method returning an enum, plus two nullable getters, getText(): ?string and getFile(): ?File. Only one of those two is ever populated on a given part, so checking which one isn’t null tells you exactly what you’re holding:

foreach ( $result->toMessage()->getParts() as $part ) {
    $text = $part->getText();
    $file = $part->getFile();

    if ( null !== $text ) {
        echo wp_kses_post( $text );
    } elseif ( null !== $file && $file->isImage() ) {
        echo '<img src="' . esc_url( $file->getDataUri(), array( 'data' ) ) . '">';
    }
}

That’s the pattern this chapter actually uses from here on.

Seeing it work

Create chapter-14-multimodal-output.php inside includes:

Add [ai_course_ch14] to a page and load it. Instead of a wall of text or a single picture, you get something that reads like an actual illustrated recipe, a step written out, a photo for that step, the next step, another photo.

Notice the try/catch ( Throwable $e ) wrapped around the loop. is_wp_error() only catches errors the AI Client reports itself, not a genuine PHP error like calling a method that doesn’t exist.

That kind of error is a Throwable, not an Exception. A plain catch ( Exception $e ) wouldn’t catch it, Throwable is the one that does.

Try it yourself

Write your own prompt that naturally wants both text and images together, a short how-to, a product description with a mockup, a story with an illustration. Run it through this chapter’s shortcode and see how the model splits its response into parts. Count how many text parts and how many image parts come back, it won’t always be a perfectly even alternation the way the recipe example suggests.