(bilibili link: https://www.bilibili.com/video/BV1uP411b7Fz/)

Our studio, Spellbrush, makes two core products: niji・journey, a generative AI model in collaboration with Midjourney, and Arrowmancer, a videogame.

You may wonder now: why two products? A videogame and an ML model are two completely different things. In reality, they’re quite symbiotic. We test all the niji features in the Arrowmancer production pipeline, before we ship them to you.

Recently, it occurred to me that as a result, this game is a record of the history of generative AI. Here are some shots from the archives.

Arrowmancer 2022 Summer Event

Arrowmancer 2023 Summer Event

What a glow-up!

It all started back in 2019, before niji was even a spark in our synapses, before anybody realized Gen AI would grow into the behemoth it is today.

Back then, the landscape of generative AI looked very different. For one, text-to-image research wasn’t mature yet. Imagine an image generator where you couldn’t control the output!

Arrowmancer is based off our Waifu Labs engine, where we used vector decomposition on the latent space of generative adversarial networks to drive character design. That’s a bit of a mouthful: You can envision roughly what that means with the following video.

While the picture results were hard to control (imagine prompting in math equations, instead of words!), they transformed beautifully:

Same character, different pose:

Same pose, different style.

And so we designed a game where you could take your very own creations from the Waifu Labs engine, and change them through your adventure.

These image results were very high quality for their time (and remain passable, as of this writing in September 2023), but there were some insurmountable limitations:

So we crafted the mechanics of Arrowmancer to fit the unique constraints of the technology: In the core gameplay, the player used their fingers to manipulate bodiless headshots on tiles.

Since the only way to control the Waifu Labs engine was with two existing pictures, we made the metasystem revolve around the concept of character merging.

Given the limitation that new pictures were made from merging existing pictures, players found humorous ways to express their creativity,

After 2 years in development, Arrowmancer opened its doors in the January of 2022, with what was essentially the cutting edge, state-of-the-art in image generation at the time!

Just 6 months after Arrowmancer’s launch, immense changes to the field were on the horizon. A new technique, CLIP-Guided diffusion, had been pioneered. At the time of its inception, It didn’t match the quality of the GAN images, but it offered a whole new layer of control: instead of stumbling blindly in the dark, you could now turn text into images.

This is the technology that powers both niji・journey and midjourney today.

For the first time, we could produce images that weren’t headshots. We used diffusion models to produce the Venus Underworld area in Arrowmancer, a moody underground city of corruption and crime. (very midjourneycore)

Diffusion models weren’t good enough to draw faces back then, but we could use them to make good backgrounds.

To bring these scenes to life, we drew characters on top, to produce in-game comics for Venus Underworld.

At this point in time, we thought that maybe generative AI would never really get to the sophistication of making character bodies, but it was still really useful to have the backgrounds!

Really, what an outdated thought.

In a mere month, model development was proving our predictions wrong. These are images from arrowmancer’s 2022 summer event, set on Jupiter’s balmy ocean moons.

I look back on these bad drawings with great fondness: It was a magical start to the model that would eventually become niji v4.

These tiny picture were the highest resolution of the proto-niji model, clipped straight from our test IDEs!

At this point in time, GPT was growing into the public consciousness. In particular, we were curious about how we could drive prompts with large image models.

For Arrowmancer’s Halloween 2022 event, we made a huge maze in a non-manifold house, with custom descriptions and bespoke images, generated via large language model. Each room was uniquely furnished by GPT-3.

Which the players enjoyed greatly:

This Arrowmancer event was released just one month before the meteoric rise of ChatGPT!

By December of 2022, we had stabilized the style of niji V4. Some of you may recognize as the first picture of the daily themes channel. To complete the final tests for niji, we used the niji model to produce all the illustrations for the Arrowmancer Christmas assets.

At this point, the infrastructure of Arrowmancer was getting hard to maintain, in light of all the advancements in generative AI. We explored ways to bring the characters in Arrowmancer into different media, so we could continue testing more experimental interactions.

We took the most spirited character from Arrowmancer, and invited him visit the niji server for April Fool’s Day. Many of you might have met Blk Cat the black cat, during the day he was poking around #sekrit-channel.

He tried his best to tell everybody niji v5 was gonna be great.

His backstory inspired some discourse

And seemed to have made a lot of friends:

Really, who wouldn’t love this handsome fellow?

With the development of the panning feature, we streamlined the Arrowmancer character production pipeline. June 2023 had us once again in Jupiter, producing another summer level.

We had a backwards-compatibility problem at this point: Every single character in Arrowmancer is mixable with every other character, so this meant that all characters must exist in the space of Waifu Labs. (In order to perform the latent space mixing computation)

Before, when Waifu Labs was state of the art, this wasn’t an issue. Now, with the release of Niji V5, this was proving to be a problem, since niji was overtaking the quality of Waifu Lab’s headshots: in-game cutscenes with niji were getting hard to match to the the lower quality Waifu Labs style.

Upon the development of panning, we could now extend a Waifu Labs picture. We used it extensively to test niji’s panning and zooming features, before its release.

Taking the original headshot

We use niji to pan out on the head, to create character designs.

Some parts of the panned images are used directly:

While other parts are used as inspiration for a human artist:

The character is then cut:

and animated!

We’d come a long way from headshots of characters on tiles.

During July 2023, we were producing the last Arrowmancer update. Here, we heavily tested the inpainting feature.

In terms of game production, inpainting is an IMMENSELY useful technique: Here’s a prototype for different costumes:

Different facial expressions:

Continuous cutscenes:

We made extensive use of all the techniques we’ve researched so far to make the final chapter of the Arrowmancer saga beautiful!

We’ve come full circle, back to headshots: Even though it is just a simple headshot, somehow niji’s rendition speaks volumes, in a way that Waifu Labs never achieved.

Today’s game paradigm is strange. Traditionally, a game is launched when its content is complete. Nowadays, a live service game ends when the content is complete.

Today, we completed the Arrowmancer story.

For the ending song, we celebrated with some lovely human-created art

Which turned into the music video you saw at the top!

In a world of infinite anime pictures, the audience ultimately responds to the “concept.” It’s far more than a character’s image data: it’s what he thinks, he eats, what he does on his free time. We have all the images in the world, but mere images don’t make a game: modern gaming is a full-bodied experience which combines pictures, gameplay, story, and much more.

One of the strongest pieces of the Arrowmancer experience is its soundtrack.

https://www.youtube.com/watch?v=syF7t7gQn68

I attribute this to the overwhelming skill of the musicians on the project. Together, Kirara Magic and Shion have created an exceptional brand of future bass for our soundscape, a soothing female vocal layered on top of EDM beats.

It’s really delightful, and I highly recommend both of their channels, if this is your jam:

Kirara Magic: https://www.youtube.com/c/KiraraMagic

Shion: https://www.youtube.com/channel/UCri7Aft0jr2Jd0wIdTft84A

As well as our lyricist, Akira-san, who once told me that the secret to writing good lyrics is “fragrance”

Akira Okumura: https://twitter.com/okumura_akiiira

The final song of Arrowmancer featured some special guest appearances by vocalist Rachie: https://www.youtube.com/channel/UCRbsmnDTHD9-4pEIuG7zYRw and lyricist Ashe: https://www.youtube.com/user/ashestoashesjc Both of whom gracefully put up with my selfish childhood dream request of a duet in both Japanese and English, which complicated logistical matters to no end.

We’ve pushed the content delivery system in Arrowmancer as far as it can go. In order to travel beyond this point, we have to engineer a new system designed for niji’s specific set of constraints.

Arrowmancer is complete, but we’ll continue this tradition of moving an ML model and a videogame in lock-step with each other, exploring the frontier of the interesting experiences that can be made with generative AI. I’m taking a break. Then we’re starting work on our next game.

As always, thank you for supporting us on this journey!

Related Articles