Generative AI is a field of artificial intelligence (AI) that focuses on systems capable of generating new content, such as images, text, and audio, that mimic human creativity. Unlike other forms of AI, such as machine learning, which focuses on analyzing and making predictions based on existing data, generative AI creates new content from scratch.
One of the key tools used in generative AI is the neural network, a machine learning algorithm designed to mimic the function of the human brain. These networks are composed of many interconnected nodes capable of processing information, and there are many exciting applications for these networks in creative fields such as art, music, and literature.
Some examples of generative AI are deepfake videos, which make a recording appear as if a person is doing or saying something they otherwise haven’t, or AI-generated music, which can break away from traditional structures.
Society is still deciding how to integrate this technology. Until then, generative AI remains an exciting and rapidly evolving field with the potential to transform how we create and consume content. In this article, we’ll explore ways these tools can boost your creativity.
Creative Uses of Generative AI
Generative AI uses various methods to produce different types of content – from art, music, and text.
By understanding these different techniques, you can develop better prompts and results. If you’re completely new to AI applications, check out our Introduction to ChatGPT course.
AI and Creative Writing
Poetry and literature
Natural language generation (NLG) is a technique in generative AI that involves generating text which mimics the structure of human language. This technique has been used in generating new works of literature and poetry.
The basic idea is to train a neural network to learn the rules and structure in order to generate new text that is grammatically correct and stylistically consistent with human language.
NLG has been used by writers and poets to create new works of literature and poetry. Like with other media, an author could use NLG to generate a novel which incorporates elements of existing works, only with a unique plot and characters. A poet could use NLG to generate new poems that explore different themes and styles of poetry.
The Aum Golly series of poetry books is an AI commentary on humanity, generated by Finnish publishers in 2021 and 2023. Fully illustrated and available for purchase on Amazon, the creators have been completely transparent about the cost and time required to create the works: “24 hours and $2.64” in the instance of the first installation.
While poetry may be a bit more accessible for machines still working on their ability to mimic English grammar, novels and longer writing has proven to be a challenge for NLG. In 2017, writer Ross Goodwin attempted to use AI to create a version of On the Road, the novel by Jack Kerouac. Goodwin published the resulting 1 the Road for research purposes, despite it being choppy and largely unintelligible.
In addition to literature and poetry, NLG has also been used in other creative applications. For example, it has been used to generate product descriptions and marketing copy for websites and e-commerce platforms.
Chatbots and virtual assistants
Chatbots and virtual assistants use natural language processing and machine learning algorithms to simulate conversation with humans and provide assistance in various tasks.
In the art world, chatbots have been used as interactive exhibits in galleries and museums. Visitors can ask the chatbot about the exhibit or artist, providing a more engaging and interactive experience.
Virtual assistants, on the other hand, have been used in creative industries such as advertising and marketing. These assistants can help companies create and manage their social media accounts, generate content ideas, and interact with customers.
For example, using a virtual assistant to respond to customer inquiries on social media helps build a more engaging relationship with customers while managing the intense workload.
In addition to the art and creative industries, chatbots and virtual assistants have also been used in various other applications. For example, Woebot is a chatbot that uses cognitive-behavioral therapy to help users manage their mental health.
Sentiment analysis and text classification
Sentiment analysis and text classification techniques are used to understand the sentiment or category of the text. These techniques have been used in various creative applications, including generating new works of literature and determining the emotional tone of the text.
You could train a neural network to identify whether a piece of text is positive, negative, or neutral in tone. Writers could then use this to develop new works of literature which evoke a particular emotional response.
Text classification, on the other hand, involves categorizing text into categories. For example, you could train a neural network to categorize articles into different topics, such as politics, sports, or entertainment, which can then be used to generate new articles which explore similar topics and themes.
Artists and writers have used sentiment analysis and text classification to generate new works of literature that explore different emotional tones and themes.
Tips for AI and creative writing
If you want to use AI to help with your written creativity, make sure you get familiar with prompt engineering for tools such as ChatGPT and Google Bard. To generate high-quality results, you can guide generative AI models by providing detailed instructions. For example, in the case of GPT-3.5, you could ask it to write a sonnet about spring, or to continue a story in the style of a certain author using a specific prompt.
Check out our ChatGPT cheat sheet for more details on using the right prompts.
AI and Art: Still Images & Video
Generative models can accept both static and real-time data, like artist Rafael Lozano-Hemmer’s installation called “Pulse Room,” which uses generative AI to create a unique light display based on participants' heartbeats.
The installation uses sensors to detect the participants' heartbeats, and generative algorithms create a unique pattern of lights based on their heartbeats. We’ll cover these and many other techniques you can expect to use with various open, generative AI tools for digital media.
Style Transfer is a technique that allows an artist to apply the style of one visual to another.
The basic idea behind style transfer is to train a neural network to recognize the distinct style of a particular artist by feeding the network a large dataset of images with the desired style. It then learns to identify the characteristic features to apply to other images.
The network then takes two images as input: a ‘content’ image and a ‘style’ image. Applying the learned style to the content image, a new image that combines the content of the original image with the style of the ‘style’ image is generated.
Artists have used style transfer to create new works that combine various artists' characteristic styles. For example, an artist could use style transfer to combine the style of a Picasso painting with the content of a photograph. This would result in a new image with the characteristic features of the Picasso painting and the photograph.
The Mona Lisa in the styles of Picasso, Van Gogh, and Monet
Image-to-image translation is a technique in generative AI that allows an artist to translate an image from one domain to another. For example, an artist could train a network to map images of horses to images of zebras. The network would take the characteristic features of each domain and could then use this information to translate images from one domain to another.
The most popular example of this technology might be the craze over the Lensa AI app on social media starting in late 2022. The app essentially takes your selfies and turns them into digital avatars replicating various artistic styles.
Similar versions have emerged all over the internet. For example, Media.io gives you a bit more control by allowing you to choose the image style you apply to any selfie – from Barbie to comic book, and animated caricature – it’s clear how image-to-image translation works on this app.
In addition to art, image-to-image translation has also been used in other creative applications. For example, it has generated new visual content for websites and social media. By translating images from one domain to another, companies can create diverse visual content that appeals to a broader audience.
Through this process, the generator learns to create images that are increasingly realistic. Artists can use GANs to create new images inspired by existing works of art, or create entirely new images.
Data augmentation is a technique that applies transformations to existing artistic material. This process helps create new inputs, which can then improve the outputs of a neural network. The basic idea is to improve the accuracy and robustness of the network, leading to better results.
There are many types of transformations that can be applied to existing data: rotations, translations, or reflections. Changing the brightness or contrast of the image, application of filters, and more, are all variations which could create a robust set of artistic inputs. Once the transformations have been applied, you can use the resulting images to train a neural network using the resulting images. Then artists can create new work incorporating elements of existing pieces with unique variations and transformations.
Artists have used data augmentation to create new works of art that explore the boundaries of styles and techniques. For example, an artist could use data augmentation to generate a series of variations on a particular style of painting, exploring different color palettes, brush strokes, and compositions.
One example is the collection by Mario Klingemann, “Memories of Passerby I,” which is trained on a dataset of portraits from the 17th and 19th centuries and augmented with transformations of these images.
Tips for AI and art
When using AI tools in art, remember that they are tools to enhance your creativity. It's essential to experiment and iterate with different inputs, parameters, and styles to create a work of art that aligns with your creative vision. Ultimately, the art you create should be an expression of your ideas and imagination, with AI serving as a powerful tool to bring those visions to life.
AI and Audio: Sound & Music
Musicians are also using generative AIs to create new music. For example, neural networks can take existing songs and identify the patterns and structure of the music to generate new melodies and chord progressions. This tech would allow any non-musician to create new music inspired by existing works while still being unique.
Algorithmic music composition
Another use of generative AI in music is algorithmic music composition. Generative music can be created in real-time using algorithms programmed to be unique each time it is played. One such example is the app Endel, which creates generative music based on the user's preferences, location, and time of day.
Other examples include the generation of a series of melodies that all follow the same chord progression or rhythm. The resulting music can be used as a starting point for a new composition, or combined with other elements to create a complete piece.
Many algorithms can be used for music composition, each with its own strengths and weaknesses. For example, some algorithms generate music based on rules or patterns, while others use machine learning techniques to learn from existing compositions and generate new ones.
For example, translation models allow an algorithm to gather information from nonmusical media (such as an image) and translate it into a musical result. One tool you can experiment with is CLIP Interrogator, which develops a brief track or loop in response to a single image you have uploaded.
Music style transfer
Like in other mediums, style transfer allows for the application of key artistic variables to different pieces of music. This technique has been used in several creative applications, including developing new works of music and remixing existing tracks.
For example, a musician could train a network to map classical music to electronic music. The network would identify the characteristic features of each style and could then use this information to transfer the style of one piece of music to another.
In addition to music, style transfer has also been used in other audio applications, such as speech and composition. By transferring the style of one piece of content to another, creators can explore new ways of expressing themselves and challenge the boundaries of traditional forms of media.
Generating sound effects and ambiances
Sound generation using AI has been a growing field in recent years, with applications in music, film, and gaming. By training a network on a large dataset of sound effects, it can learn to generate new sounds that are similar in style and texture. This technique can be useful in various creative applications, such as creating sound effects for video games or adding ambient sounds to film.
Another approach to generating sound effects and ambiance is through the use of procedural audio. Procedural audio is a technique that involves generating sound effects in real time based on a set of rules and parameters. This allows for highly customizable and adaptive sound effects that can change dynamically based on the context of the project.
Tips for music and AI
Harnessing AI in sound and music unlocks expansive creative potential. From generating original compositions that blend various styles and moods, to the exciting possibilities of audio style transfer and crafting immersive, personalized ambient soundscapes, AI offers myriad possibilities.
It also extends its prowess to automated mixing and mastering for high-quality results and generates unique sound effects to establish a distinct sonic identity. As a digital muse in songwriting, AI can provide a wealth of ideas for melodies and harmonies. Remember, these tools are here to enrich your creativity, not replace it.
Final Thoughts & Resources
The creative process can, at times, be difficult. However, with generative AI, anyone can get inspiration for their projects. Similarly, using the tools and techniques we’ve outlined here, even established artists can expand their work at scale.
While society continues to explore how this technology will affect culture and impact our future, it’s a great moment to understand these tools and develop your own opinion.
Here are some resources to learn more about and stay up-to-date with developments in Generative AI:
A data science and public policy graduate student at the Hertie School and a writer covering tech, AI & social issues.
The Top 16 AI Frameworks and Libraries: A Beginner's Guide
Is AI an Existential Risk? With Trond Arne Undheim, Research Scholar in Global Systemic Risk at Stanford University
Building Human-Centered AI Experiences with Haris Butt, Head of Product Design at ClickUp
Weaviate Tutorial: Unlocking the Power of Vector Search
CoCa: Contrastive Captioners are Image-Text Foundation Models Visually Explained