AI's "Just Ship it." problem
Throwing spagetthi at the wall faster doesn't make your wall or audience bigger.
Do you, a B2B SaaS product, want to sponsor this spot so it’s free forever and get your message out? Drop us a line for the Mediakit: sponsoring@productea.io
"Let's fire everyone who's junior and midlevel. We have AI now!", is AI really boosting everyone's productivity to the wazoo, or is it shifting bottlenecks?
Everyone who actually does the work knows that while AI gains are impressive across the board, they are not as crazy as we might think.
The other day, I watched a video titled “Does AI actually boost developer Productivity?” by Stanford, conducted on 100,000 developers.
It attempts to ground the real numbers in terms of productivity gains, specifically for developers. There are a lot of parallels, though, to other jobs as well, including product and growth teams.
I joined Smallpdf back in 2019 when we were some of the best at throwing things at the wall to see what stuck. We were about 20 highly talented people with a strong engineering and shipping focus.
It made the company successful and grow at a breakneck speed. I also remember what my CEO, Dennis Just, told me on the first day of work:
“We’re really good at throwing spaghetti at the wall and seeing what sticks, but we need to grow out from that; it’s become unmanageable, and growth will eventually stall.”
How right he was.
Let’s look first at some brief numbers from the study and then at the parallels to what it means for companies and their products, and why the term “shipping” is misunderstood.
"As codebase size increases, productivity gain from AI decreases"
This is also true for other jobs. The meaningless, "simple" busywork can be massively improved with AI but complex work still suffers from being something an AI cannot comprehend.
We're looking at a range from simple, 60% to complex, 5 % more efficiency:
These numbers are in a field (engineering) that is “relatively” measurable. The worry I have is the usual lemmingesque behaviour of CEO’s not understanding any of it and translating numbers they hear on podcasts from “massive increases” to “I think we can fire half of our staff”.
“Using AI in software engineering increases rework”
You're delivering more and shipping more. Not all of it is useful; not only is that stuff also requiring rework. It usually requires rework that AI can’t solve by itself.
This is especially the case when we talk about complex system integration, where you have to make various systems work with each other.
The spaghetti effect and “shipping”
The relative position you inhabit in a market is, to a large degree, influenced by your competition’s capabilities. And in 2019, Smallpdf was well ahead of its competition. We dominated our distribution (SEO) and were shipping things multiple times a day.
Throwing spaghetti on the wall to see what stuck and reiterating on it led us to success with impressive numbers, hypergrowth at a scale that most companies and funds dream of.
At its height, we had 50,000,000 monthly active users using our services to play around (worked on by about 10 teams, product & growth). This traffic was divided into the few dozen tools we had, which was impressive by any standard.
And while this was by far the biggest data stack I’ve worked on at that time, I still ran into problems at scale:
The bottleneck of the team was waiting for statistical significance for most of our experiments, despite all the traffic we had.
Shipping a lot of small crap without validating it is the same as polishing a turd. The product constantly changes, but not necessarily for the better.
The company was struggling to come up with big, good swings that were worth building. Should we add editing of a PDF into the product, or make all tools collaborative? Should we ship less so there’s less disruption to the service? Should we rework the legacy code that has not been touched in 4 years for most tools?
Especially the first point was amazing to me. If we changed a copy of our main website or most prominent tools, the experiments were statistically significant within hours.
A more complex down the funnel change for higher value customers? Weeks, sometimes months.
Ugh.
If we had access to AI improvements and faster devs, none of that would have changed. You can ship reiterations faster, but I remember specifically that my planning was constrained by how many tries I had to improve things with the team per given week.
If you haven’t yet, you will run into the same problem. Regardless of how many people you reach.
The term “shipping”
“Just ship it” was the go-to phrase of a manager I used to have over 20 years ago. It annoyed me to no end. He constantly said it because he felt that everyone was slow and not keeping up with his great ideas.
While I developed some sympathy for why he said it (he was convinced that he had the best ideas), it highlights a typical management problem. Managers (especially sales-led ones) don’t understand what “shipping” means and usually forget about a key part of it:
Ideation: There’s an idea
Development: You build the idea
Validation: You validate whether what you think the idea does is actually happening
Yes, vibe coding tools like Lovable et al. help you to ship things faster, but only as long as these ideas struggle with the “development” part and don’t need Ideation and Validation:
AI doesn’t help you come up with better ideas, except for the most obvious ones. It can help you with processing feedback somewhat though.
AI can’t help you to validate anything at scale faster. Validation depends on how many users you have to test on, not how fast you “read” the results.
Now, you might be a manager who rightfully points out:
“Yeah, but Leah, I know what we should build, and we will see in the sales numbers whether the ideas work. I don’t need analytics for that.”
That’s true, only for the smallest companies, if you have a lot of obvious things left to build and improve.
As you scale, this does not work anymore because a good idea might give you a lift of only 5% which is undetectable in a typical sales pipeline.
You have one product, and we know that 75% of ideas, despite their best intentions, do not deliver any meaningful impact. Even worse, they might negatively impact the overall product and customer experience.
By skipping the validation step, you might ship yourself literally out of business.
Conclusion
If AI improves the facilitation of smaller changes, then these will not be a bottleneck anymore.
What’s left will be complex problems and validation. Problems that AI can help with very little.
Building a tech and software product that is commercially viable will always be hard and more of an issue of Ideas that matter and how you distribute them.
If it wasn’t, it would be copied, and you would drown in commoditized offerings, and you can’t charge money anymore for it.
Yes, you should absolutely think about how to integrate AI into your workflows if anything, and enable your organization to work with it, but chill on your budget because of it.
It’s great, but not that crazy to just forget about everything else.
Talking to customers and building the “right” thing for the correct market is still king. So far, AI has not substantially improved or made any of it obsolete.
Thank you for writing this. I actually disagree with a lot of the "PRDs are dead" commentary we're seeing online; the thinking is precisely along the lines of your post (just ship it, don't think about it!).
I wrote this post with Ravi, which looks at things in the other extreme: seeing specs as the source code: https://blog.ravi-mehta.com/p/specs-are-the-new-source-code
Would love your thoughts :)
Love this breakdown, Leah - especially the reminder that velocity without validation is just noise. AI might speed up the 'how,' but it still can’t replace the 'why' behind great product decisions.