My Year in Data

Over the last year, I have had 9,497 exchanges with an LLM: 4,763 from me and 4,734 from Claude. After AGI is the first production on which I've used AI in such a comprehensive way. In this chapter, I will explain why I chose to use AI in different parts of my creative process and what I learned over this year-long study.

As a one-person studio, this was also the first time I've created in so many fields at once. AI definitely made this possible, and I collaborated with it on most of them. I broke this process into 18 jobs, including scriptwriting, songwriting, music production, front-end software development, back-end software development, choreography, dance, film, editing, art design, publishing, UI/UX design, and graphic design.

My skill level across all of these was not equal, and neither was AI’s. As I discuss my findings in this chapter, I will reference these 4 variables as they pertain to each job: my skill level, AI’s skill level, the ratio of human-AI involvement, and my desire for personal growth (or meaning).

You vs. the Machine

You vs the Machine 01 (Figure 16) lays the foundation for how I compare to AI when collaborating with it across various domains. It shows the eighteen jobs ranked by how skilled I was in each. My skill level is calculated from years of experience, which can mislead. For example, in cinematography, I started personal projects eight years ago, but I wasn't working professionally in any of those years. I still found this to be the most straightforward representation of my skill level for each position.

AI’s skill level at each job is also a subjective metric in Figure 16. AI benchmarks are frequently updated, and a field like cinematography is not given a separate value in these standards. I estimate this value based on my experience using AI in these fields and my confidence in AI to automate these roles. These numbers might be different for you. So I encourage you to make your own; this can help you understand where AI might be most useful to you, whether for learning new skills or automating parts of your production process.

All 18 jobs in descending order, with the fields I had the most years of experience in at the top, are shown in Figure 16. Performing professionally as a ballet dancer since I was a teenager placed Performer at the top of the list. Although virtual avatars are on the rise, I determined AI was not very skilled in this area, nor did I ever want to use AI as an actor. I placed its skill at 0, which is as much a statement of what I wanted more than what was possible. I had zero experience in music production and songwriting, both of which I had never done before this project.

For many of the jobs where I had the least experience, AI was, by contrast, very capable. These include scriptwriting, songwriting, and music production. You vs the Machine 02 (Figure 17) shows an alternative view of this comparison. All the jobs in the left triangle are where AI was stronger than me. In the top-right corner are the fields where both AI and I have significant skill. There might be a time when AI is more capable than me in every job by some standard. Yet, I wouldn’t choose to automate everything. This is where the variable of personal growth (or meaning) factors in.

Automation Vs Personal Growth.

Although AI was more capable in Music Production, where I had zero years of experience, I took on these tasks myself and learned the craft. Personal Growth vs. Automation (Figure 18) makes this clear across its 4 quadrants. The top right shows the jobs I decided were the most meaningful; the top left shows the jobs most tempting to automate; the bottom left shows the jobs I happily automated; and the bottom right shows the jobs I wished could have been automated more. Each job is placed on the graph as a circle; the larger the circle, the more capable the AI was at that task.

This shows that a higher desired level of personal growth in a domain led me to keep more of my human touch in the loop, regardless of AI’s capabilities. It also depicts how I might have desired more automation in areas where I already had strong skills and felt less need for personal growth. All jobs that cluster in the top-right of the graph are the ones that were most meaningful to me in creating this musical and therefore had the least amount of automation. The jobs that clustered in the bottom right were areas where I wanted more automation, but I still executed the vast majority of those tasks myself. Jobs that fall in the middle have a mix of both. Ideally, the jobs in the middle of this plot would be jobs you have high skill in, as does AI. In this case, the creativity of the machine and human can augment each other, which was the case for me with Front-End Development and Research.

What these suggest is that in creative work, setting intentional areas for personal growth, or where the meaning of your work lies, is critical. This value can guide how far you explore AI as a co-creative partner across all phases of a co-creative loop. With less knowledge in a field but with a strong desire for personal growth, AI could help you gain the skills you are looking to achieve. However, since you are less knowledgeable in this area, it's good to read what experts say about where AI might fall short in this domain, helping you gain a better understanding of where you can rely on it.

If you have tremendous skill in an area, there is ripe potential for you to explore how technology augments your creativity in ways that are personal to you. You might want to continue expanding your skills. When you and the machine are both highly skilled in a domain, it can become an unparalleled creative partner. For example, through research and front-end development, I determined that these were areas where the machine and I both have strong skills. This harmony makes speaking with the machine seamless, and with the speed at which AI can work, it will tremendously enhance what you can conceive of and produce compared to working alone.

Trends & Findings

You vs. the Machine and Personal Growth vs. Automation figures both visualize why I chose to automate some jobs more than others. But does the data — my actual conversation history — reflect the same thing? Figure 19 shows all of my messages per month with Claude.

Across the three built artifacts, the web app, the musical, and the subway installation, the signatures diverge. The majority of my conversations with AI were regarding the web app. Most of my conversations with AI concerned the web app: long threads in condensed sprints. The musical, general research, the script, and the score produced much shorter but more consistent discussions across the year. I talked to Claude about the subway project only at the beginning, while researching the project and in the months leading up to the published campaign.

When we look again at my co-creative loops, the website is a knot of small, fast loops: front-end and back-end work I was collaborating on with AI, turning dozens of times a week during the sprint period in October. The musical is the opposite: many large loops where I refused to use AI in many phases. This provides the first evidence that the loop is not just a nice picture but a real description of how the work was made.

Volume per month tells you when Claude and I were talking. It does not tell you whether those conversations were beneficial or productive. So I tagged all 579 conversations in the record into these 4 categories: useful, mixed, went nowhere, or refusal. Figure 20 plots all of them at once. Each dot represents one conversation, placed on the horizontal axis by how long it ran, and color-coded into one of the 4 categories.

The first thing to admit is that most of them went nowhere. 300 of the 579, or 52 percent, produced nothing I used. Only 195 were useful. This includes conversations about admin and other daily life topics. For the 240 conversations about the content of After AGI, 146 were useful, and only 35 went nowhere.

The second thing is the sharper one, and it is visible as a shape before it is visible as a number. The conversations that went nowhere pile up on the left of the plot, at two and four messages. The useful ones spread out to the right. A conversation that produced something ran to a median of 10 messages and 1,621 words. Ones that went nowhere stopped at 4 messages and 452 words.

The same shape repeats by domain. Front-end and back-end development, where I ran my longest threads at a median of 30 messages, returned useful work 72 percent of the time. Research, where I typically asked once and left at a median of 4 messages, returned useful work 15 percent of the time. Music is the exception worth noticing: short conversations, a median of 6 messages, and still useful 83 percent of the time. By the time I was in music production, I knew exactly what I was asking the machine for, and a short loop was enough.

Across all my tagged conversations, AI refused to collaborate once. It came early in the project, while I was getting feedback on a song in Draft_01 of the script. Claude's reply was:

Honestly, it seemed like a reasonable response. After that exchange, I avoided using AI for similar discussions and, more broadly, I kept it out of the phases where the content was conceptual rather than technical. So this low refusal rate is less a verdict on the tools than a measure of my own caution.

The conversation data also confirms that I was speaking with AI about multiple jobs, often simultaneously. Through this process, I've become comfortable with running multiple conversation threads at the same time. This was not the case when I began. But in a co-creative practice, your workflow may change completely. Mine certainly did. And it took a real adjustment.

P11: Become an Expert in Cross-Skilling and Agent Management

You might think it is ridiculous for me to be working in 18 different capacities in one year. I won't argue with you, but when working with AI, you will find yourself juggling multiple tasks at once without even realizing it. Especially for small teams or individual studios, your job is not only to create work but also to produce and publish it. For this, I suggest the principle: Become an Expert in Cross-Skilling and Agent Management.

Crazy or not, everyone has to upgrade how they work because of AI. With this change, I predict each individual will be required to do more than one job simultaneously. But rather than being burdensome, I think this can be an incredibly fulfilling way to work. You have more agency when your hands are in multiple aspects of bringing a product to life. But this requires a new mindset. And it requires you to learn new skills continually.

“Cross-skilling” is learning skills outside of your primary domain, but bringing your subject matter expertise with you. There is much overlap between fields, which can be a comforting place to start when learning a new subject. In asking AI for assistance in learning new terminology, you can say, “Can you teach me fundamental terms of graphic design using similar terms I already know from my background in dance?” In expanding your knowledge, you can leverage your grasp of your field to map how other fields might differ or overlap with your own. A good place to start is a field adjacent to your own, such as UI/UX design if you are a front-end developer.

While working in a Co-Creative manner, you will find yourself speaking with multiple agents at the same time. A common cause is a significant delay between responses while an agent is executing work. You could be managing just 2 agents, or maybe you are fantastic at management and can handle up to 20. It doesn't matter how many, as long as this number works for you. For me, when I attempt to manage more than 5-6 agents, I start to lose track. The data on multitasking suggests this is an ineffective way to work. There is loss of productivity when toggling between tasks (Mark et al. 2008). But when managing multiple agents, your role is not one job; it's being a manager, planning tasks and deliverables for each agent.

But be careful of “Token Maxing.” This term has arisen because there is pressure to run multiple agents at all times, spending every token you have to maximize your work's productivity. It is important to note that the work you are doing isn't always managerial. So don't hesitate to stop the herd when you need to regroup and focus on a more singular task. Sometimes you just need to do one thing at a time! The opportunity cost of not taking advantage of these agents can feel real, like you are falling behind and making you obsessed with running agents while you sleep just to feel more productive. But productivity isn't the goal; the goal is the best product you can realize. I found that doing one task at a time with one, maybe two, agents was the workflow I primarily used for deep work.

P13: Intentionally Decide what the Dirty Work is.

Historically, famous artists would have apprentices who would finish details after the master had made the essential artistic decisions. In a future with multiple agents at our fingertips, it is tempting to use them not only to finish details but also to complete entire portions of work, or even entire jobs. Reasonably, the most tempting is to automate tasks you’d rather not do. But those tasks aren’t always best suited for automation.

Tagging this set of conversations, for example, would have been easy to hand off entirely. Complete automation is consequential, so you must be aware of its impact on your overall result. Use AI where the algorithm can work not only faster but more accurately than you, then check the output anyway. For classifying data as I was in tag conversation transcripts, I went over every conversation myself, because it mattered that I confirmed which of the four categories each one fell into.

Determining which jobs to automate, not out of pure convenience, but for the quality of your work, is a skill you need to develop in a co-creative practice. Judgment of when to use AI and when not to is a capacity we will continue to update with new model releases. When using AI, intentionally decide what the dirty work is, with an informed perspective of what AI can execute properly. Keep in mind that delegating work to an AI should improve the quality of your work rather than deteriorate it for the convenience of saving money, time, or other resources. The quality of your work is all you have once your project is finished. You want to be able to stand by what you’ve created with confidence, knowing it is the best work you can make. Or the best work you can make augmented by AI.

What I would have done differently

Frankly, working with AI took up a sliver of my overall time on this project. But the data I recorded for this process was mostly just my interaction with Claude. I spent so many more hours working without AI on all aspects of this work, from the script to rehearsing, filming, and editing. I didn't have a system for documenting my work away from the computer. This would be an important addition to my methodology in the future.

I'd also like to design better metrics for assessing when AI's involvement was counterproductive. I tracked refusals, but I never tracked, in real time, the moments where a conversation felt confusing, discouraging, or simply wasted. Occasionally Claude would ask me, "How am I doing?", Anthropic trying to source data of their own. I ignored these every time. I wish I'd built a tracker to record when AI felt like a hindrance while I was in flow, which would have given a far more detailed picture of which roles and subjects Claude served worst, and why.

Lastly, if I had had more time, I would have delved deeper into the psychological motivations behind creative work and creatively inclined people. When determining the creative project, there is a whole list of motivations to consider. It would have been great to have written about these motivations up front to guide my interactions with AI and assess how this collaboration drove toward or strayed from those motivations. In this chapter, I introduce the idea of personal growth, or “meaning,” as a value I was striving toward. Having a more technical approach to assessing how a co-creative process moves an individual toward their goals or motivations would have been a significant addition to the psychological assessment of this process.

In the future, I plan to run a study with 10 or more participants, each of whom is given unlimited AI credits for one year. Equipping them with these principles and framework, I would like to evaluate the effectiveness of these tools in helping others create transdisciplinary projects using AI over a similarly year-long time frame.