Build Software Products for the Age of AGI

Petr Baudis Petr BaudisDecember 9, 2024

It is now a popular saying that just as software is eating the world, AI is going to eat software.

What should product builders do before their software becomes the prey? We now live in a world where AI is not powerful enough to “eat” their product, but that future seems inevitable. Still, the builders want to build something lasting. (And the investors want to invest in lasting, compounding value.)

Let’s run with a bold assumption—that AI capabilities will keep following the current scaling curve and lead us to an AGI in the next few years. And, let’s explore it on the example of a particularly inspirational domain—the wonderful world of enterprise B2B SaaS.1

What’s the AGI impact on lasting products in a world where technology is easily reproducible? How much time do we have before AGI is here? How long do we have after it enters the world? And what then – will AGI want to use any software products? Perhaps the future may not be bleak for software products that get it right, after all.

On AGI

What is Artificial General Intelligence (AGI)? For our purposes, consider a smart high schooler in a closed room with just an internet connected computer and keyboard – maybe a mouse. What they can do, AGI can do, several times faster and at roughly similar cost as the food and lodging of your indentured high schoolers.

Current ChatGPT-esque state-of-art AI2 is way ahead of smart high schoolers3 in some aspects, but far behind in others.4 Both can be very gullible and invent facts, but it violates people’s expectations more to see exacting, stone-cold machines make stuff up, rather than awkward teenagers (that they have also once been).

The mid-point estimate of many AI researchers in the top industry labs is that we may get from the current level to AGI around 2027.

Of course, you might have just raised your eyebrows sky high when reading this, but the exact year doesn’t matter, even the fact it’s not certain shouldn’t matter. Even if we say “many vocal (but real) experts in the field feel that AGI in the next 10 years is quite probable,” it’s clear builders (and investors) should have a clear plan to deal with this scenario. Even if you are building “AI products,” the GPT(N+1) launched unexpectedly on a random Tuesday evening may just push you into irrelevance with the snap of the finger. Are you going to take that risk?

On technology moats

Well, if a San Francisco AI lab can ship AGI at almost any moment, where can we find the safety we need when building a lasting software product?

Not in a technological advantage, that is for sure.

The fact is that computational intelligence is getting vastly commoditized, and in turn commoditizes any software gadget it touches. The major LLM providers are in a mad race to the bottom, with usage costs falling ~10x year-by-year5 and in tight lockstep in terms of general intelligence capabilities.

Anything that can conceivably be solved by incrementally better versions of current LLMs is not worth spending time building.

And any cool app that can be copied will now be copied much faster than in the past (when competent software talent was the main bottleneck). If the only differentiator is “we are the only app that was built for this—so far,” that’s not going to help anything.

Technology may still be worth investing in for two reasons:

  1. The investment is sheer effort required, and outside of the current LLM sweet spot. If you are building something like a database engine, it takes a huge laborious effort to squeeze out the optimal performance (combined with some ingenious ideas, perhaps non-obvious to others), and a lot of attention to detail and perfectionism to make sure it never loses data. Original thinking is not the best forte of LLMs (or most highschoolers). And sheer effort to replicate the breadth of the invested labor still makes for a defense – if we aren’t talking about a giant CRUD app but something that actually needs a lot of attention to detail, perhaps because it needs to scale well.
  2. The investment is AI capabilities that general LLM vendors aren’t going to care about. You should still go ahead and build an exciting AI technology, as long as it goes in a sufficiently different direction than what mainstream generic models provide. (Please do not take this as advice to jump on the “agent” hype.)6

To give a concrete example, at Rossum.ai we extract data from business documents as part of our core value proposition. To make it useful for our customers in their business process, the product needs to:

(a) instantly learn from user feedback on specific data fields and page layouts,7 and

(b) intelligently and auditably decide when it is confident about its prediction and when it should escalate to the user for double checking.

This is not something we see LLM vendors care about! That’s why we decided to bet the product on building our own “T-LLM” model variant that reframes the entire machine learning task differently from typical LLMs. It provides the capacity to solve (a) and (b) in quality that you won’t replicate well with, say, OpenAI API wrapper – even in 3 years.8

To further the example – we added two extra ingredients to our T-LLM. We use the generative model to make discriminative predictions about existing content rather than generate new data, though we keep in mind that general LLM vendors also focus on increasing reliability.9 We also fine tune on domain-specific datasets, which gives us some head start against general LLMs that also strive for better all-around knowledge.10 These extra ingredients are big market differentiators for us now, but if I had to bet, solving the problems (a) and (b) above will provide more lasting security; they are less adjacent to the top priorities of general LLMs.

No matter what, even if you invest in technology, you should not treat it as a moat, but as a head start. Sooner or later, LLMs will come and commoditize it anyway. For lasting value even in the mid-term, you need a better plan.

On a better plan for AGI-resistant moats

Companies with a problem don’t want to buy “intelligence”; they want to buy a solution to their problem.

The most natural product moat, or providing a lasting solution to the customer’s problem, is creating end-to-end value to a customer. That usually means better process execution (reduced effort, costs, latency) and/or better process output (increased quality and scope). That’s not a “tool” game, or something your business customers would get from “just using ChatGPT.”

If you are providing end-to-end value, chances are you have a vertical product. That means your product isn’t a component or a tool, but a platform and/or a “place”.

  1. The platform represents a unit of abstraction, in the programming sense. It encloses part of the company’s operational complexity into a simple black box that is loosely coupled to the rest, simplifying everything.
  2. The “place” is an app where an entire business process can be executed, fully covering e.g., the job description of a team of employees. Products like Salesforce, Gusto, Intercom, or Rossum are extremely sticky because they provide the place – some employees spend their entire workday in them, and they are tightly integrated across other systems and collaborating teams.11 12

While the vertical product is the obvious manifestation, it is not the actual moat. The moat is your institutional knowledge of integrating the product into real world environments.

That is: the experience and best practices (encoded across all your teams) painfully gained from delivering and rolling out the product in hundreds of cases, understanding the messy real-world needs and product requirements, the teething problems to overcome, and the surprise extra value the product provides. This moat is highly durable and disappears only in the extreme.13

The simplest external signal of your institutional knowledge is track record. You do your thing competently, long enough and in sufficient volume. That’s what generates your institutional knowledge, but it’s also something you couldn’t do without such knowledge. A flywheel.

Finally, this is what makes a brand in B2B! Brand is being a credible leader of your category – such that investing in your solution doesn’t create a risk for the company (and even more importantly for the CIO signing it off).14 Past marketing buzz,15 customers know that institutional knowledge is what really makes or breaks your solution in the real world, and look at your track record to prove you have it. That’s why it is hard in B2B to build a brand without track record as the #1 ingredient, and what makes it an endurance sport.16

As with many things in life, conquering one moat will create several others. You will find yourself with nuanced integrations that cover every corner case, and extensibility in all the right places (something that is difficult to reproduce detail by detail without real-world experience). Likely, you will bake in infosec compliance deep into your company and into your product (something that will be painful to replicate for everyone trying to follow in your footsteps).

But the main moat building in B2B is to solve all the complex, detailed, messy nitty gritty – end to end, case by case. And just going ahead with doing the “slow and boring” type of business even with fast-moving technology.

And once AGI appears?

Companies with a problem don’t want to buy “intelligence”; they want to buy a solution to their problem.

But what if companies that buy enough intelligence won’t have any problems?

…Exactly! That’s the whole point of doing AI!! In a good future (where AGI doesn’t momentarily explode into belligerent artificial superintelligence), people can stop toiling away doing things they don’t enjoy, and eventually, spaceships zoom around constructing Dyson spheres.17 But how to build a good future for our product in the all-round good future?

First, when? Let’s say AGI is launched on the evening of Sep 14, 2027.

The good future is not coming on the morning of Sep 15, 2027. Try 2037?

It takes a long time for new technology to bubble through the substrate of society, even business, and even tech infrastructure.18 Humans need to realize what happened, understand the impact, get comfortable with the risk, figure out how to transition, and then prepare countless other humans for that very same process. Then something goes wrong in the transition. AGI rollout is going to be a mega-project.19

The world’s economy relies on the collaboration of countless companies, most of which will take years to adopt any new technology, even in the face of strong market incentives.20 There will certainly be companies or even industries that AGI will disrupt quickly, and they will have to adapt as fast as they did during the COVID lockdowns. A few Fortune 500 corporations will crash and burn during this change. But the physical world is generally slow, and some complexity needs to be understood only by hard-won experience—that is not recorded in Wikipedia and Reddit training data, and that’s why AI won’t have this experience either.

On the whole, the full impact of AGI availability may well manifest only after several iterations of these rollouts across the whole economy – and take a decade. Heck, even ChatGPT impact did not properly manifest yet in the economy!21

However, your ultimate plan for lasting value needs to take into account that eventually, AGI and AGI-driven companies will become the ultimate economic drivers, replacing people and people-driven processes. So, your next question as a B2B product builder should be: How could vast armies of virtual high schoolers transform the operation of Fortune 500 enterprises, and will my product survive it?

Will AGI eat all software?

The other day, Eliezer (Yudkowski, one of the most famous AGI pundits) laughed at the prospect of future AI using primitive technology like Linux. But my hypothesis is entirely to the contrary:

Why would AI waste resources in the foreseeable future22 on creating a new OS from scratch, investing in the coordination and switching costs, etc.?

Product builders know this problem well! We call it the buy vs. build dilemma, and human buyers and builders face it daily, right now. While buying costs money (duh), the trouble with building is it costs an even scarcer resource – talent – and saps it continuously into the future.23

And AI will want to be as careful with allocating its computronium as humans are with their talent now. As e.g. Noah Smith notes, it will much rather invest in building new value only it can provide, rather than fixing what ain’t broken or reinventing what already exists. The opportunity cost of going slower in its main mission is certain to be massive (be it further self-improvement of the AI, medical research, or energy engineering projects).24

High schoolers will prefer to rely on software rather than feeding extra high schoolers. There is no fundamental reason why AI wouldn’t want to buy other software either. Even once computational intelligence is commoditized, it will always be more expensive than simple automation.

Therefore, to build lasting value, you should plan for the rise of AGI-ready products, and selling to the AGI!

A product can provide net positive value equally to humans and AGI in two obvious ways:

  1. A platform that gives any user (including AGI) easy access to all the data and actions it needs to do to fulfill its goals. Remember, the platform should serve as a simple unit of abstraction loosely coupled to the rest of the organization, ultimately so that users “up the chain” don’t have to waste their time and mental capacity on it.
  2. A place where AGI can be plugged in and the resulting virtual “exoskeleton” still brings extra value beyond just the AGI. The “place” type platforms aren’t just for people now, but for AI assistants, copilots, automations, and agents, and they can smoothly take over anything repetitive.25

Even AGI would deploy Salesforce (or a member of the post-CRM wave, such as Gong) as a unit of abstraction encapsulating the company’s RevOps, and a place where human and AI workers can meld together. We are building Rossum so that even AGI would deploy it as a unit of abstraction encapsulating the company’s execution of business transactions, and a place where (more and more powerful) AI can complement and take over repetitive document processing from human experts.

TL;DR: Do the hard thing, do it well, and have a plan

Artificial General Intelligence may come soon, but it will take a long time to truly change the world. Still, in the AGI era, technology moats aren’t proper moats and AI may destroy your product if you are not very intentional about the technology you build, and if your product is not designed as a sticky platform.

But another principle will remain constant. The moat of a brand, built on institutional knowledge gained through sweat, experience, expertise, and a consistent track record, will not go away with AGI. In fact, your technology and your track record should come together so that even AGI will want to use your product, as its computronium is better used elsewhere.

This means your product will directly help AGI bring great things to the world.

Building AGI-ready products is not magic or unimaginable sci-fi technology. It is something you can consider today and answer in your product strategy.

The best we can do is not fear the future and keep building.

THANKS to Andrej Kiska, Gwern,26 Martin Schmid, Michael Stoppelman, Michala Gregorova, Paul Akhil, Reshma Sohoni, Radek Bartyzal, Tomas Matejcek and many of the Rossum team for reading drafts of this.

—

  1. The problem of B2C is more interesting from the human-machine and network effects perspective, but also more challenging in terms of achieving escape velocity scale. We will not consider the Artificial Superintelligence (which is still a more controversial concept than AGI), or the commercial outlook and promise of building AGI itself, just assume its existence. Finally, we will discuss only the dilemmas of product builders in such a world, not other impact on society or the future of humanity overall. Good general reading is Situational Awareness, Machines of Loving Grace, and Noah Smith’s essay on comparative advantage. ↩︎
  2. Let’s call it a tie between GPT-o1 and Anthropic Claude 3.5v2 Sonnet, even though I’m in the latter camp – but anyway, “ChatGPT” is a good enough approximation. ↩︎
  3. Up to exceptions like Harry James Potter-Evans-Verres etc. ↩︎
  4. Current AI would have the edge in encyclopedic knowledge of roughly everything akin to photographic memory, or executing complex technical tasks like solving non-trivial software bugs. It will be far behind in its tendency towards trivial mistakes, inability to admit it doesn’t know, and the lack of attention to detail. ↩︎
  5. See also the Densing Law of LLMs. ↩︎
  6. First, the odds are you are not solving a clearly framed problem with a well defined reward function. Second, LLM vendors think deeper about the implications of “LLM tool use” than the vast majority of their users, and you are going to compete with them. ↩︎
  7. The walled off world of transactions between the world’s largest corporate entities is weird, and full of strange exceptions to the point of occasional absurdity. ↩︎
  8. The T-LLM provides “instant learning” experience through a one-shot learning setup that retrieves similar previously annotated pages, and is trained both to classify tokens and predict confidence scores. ↩︎
  9. “Discriminative decoder” produces tokens by just tagging suitable input tokens from the document page rather than generating new tokens from scratch.  Showing a pretty invoice to a standard multimodal model will get you a pretty demo result, but once you start measuring accuracy on long scans of scattered line-level items, generative artifacts become a problem. While a classic LLM would deteriorate rapidly,  the T-LLM by definition avoids artifacts such as hallucinations. (The same holds for prompting issues, since there is no prompt.) ↩︎
  10. Finetuning on domain-specific datasets is a natural idea since we own the whole feedback loop including data validation user experience – essential for the instant learning experience of (a). ↩︎
  11. Salesforce abstracts the “customer relationship” to the rest of the company, but also ties into the ERP, marketing automation, BI, or support desk; many customer-facing teams visit it as the source of truth. Rossum abstracts the “transaction communication” to the rest of the company, ties to the emails, ERP, or support desk, and anyone who conducts transactions outside the company visits it to approve them or as the source of truth. ↩︎
  12. We noticed that once we put the key missing bits into Rossum and made it truly a platform, our users started asking us proactively to make it a place too! Suddenly, other teams saw the emails and documents in Rossum and proclaimed that they wanted to get their approvals done in Rossum too. Once your platform achieves a critical mass, it will naturally become a place — if you have the first screen. ↩︎
  13. If your whole team falls apart at once, your product is miles behind the competition for a very long time, or the whole process disappears from the world. ↩︎
  14. OpenAI seems on the permanent verge of institutional collapse, operates on a precariously short runway, is hemorrhaging talent in all directions, and its technological edge is not an edge at this moment. Yet, its odds of winning the industry remain great – its brand is the synonym of the ongoing AI revolution for consumers. This is thanks to it being first (with a suitably captivating pinch of drama) and still executing well. (The fact that the main rival markets their flagship product as the catchy “Anthropic Claude 3.5 Sonnet newer version btw” may also be a factor.)  That’s why it can command a strong premium on its valuation, even though the investments get sunk into quickly depreciating GPUs burned down during a crazy race to minimal margins. ↩︎
  15. Everyone “knows” who the leader is and who it is going to be, thanks to hype making, analyst & consultant & orthogonal vendor endorsements, and general buzz. Some of the buzz are LinkedIn feeds, but people just talk to each other and look at what others use – the market share momentum is an extremely real moat in itself. ↩︎
  16. Once you have it, your sales team will use the track record to swoop in every time, run the account sales playbook they honed, and show up as experts in the space with all the confidence they built on top of this institutional knowledge. And then not just close a deal! but combine the institutional knowledge with a lasting web of personal relationships that will hold it in place for years to come (while playing fair – generating value both for you and the customer). ↩︎
  17. The intermediate steps are left as an exercise for the reader. ↩︎
  18. Jason Crawford writes: The steam engine was invented in 1712. An observer at the time might have said: “The engine will power everything: factories, ships, carriages. Horses will become obsolete!” And they would have been right—but two hundred years later, we were still using horses to plow fields. Sam Altman predicts the AGI moment will come and go […] and the society changes surprisingly little [just as Turing test didn’t change the society much]. ↩︎
  19. This can happen in a day in a startup, but at a large enterprise, the average software procurement cycle may easily last 6-12 months, or even more in the case of a regulated technology with well publicized risks (think: AI). That’s from an RFP process kickoff to contract signature. Then, implementation and rollout begins. Oh, well, we mean rollout towards the first (pilot) stage. Technical implementation can go slow or fast (for top-tier products that really gave IT setups a lot of thought). But getting people and teams comfortable will pretty much take several quarters in more conservative companies. Then, once the pilot is running, people can finally experience all the surprises and unexpected challenges around rolling out a totally new technology such as autonomous AGI agents – and scramble how to adapt and proceed. Years in, the real rollout only just started. ↩︎
  20. Even if an autonomous AGI-driven company arises as competition, it will need to secure or build its own production and logistics infrastructure, then do its own sales to others to make money. Did we mention the procurement process in enterprises can take months to years? ↩︎
  21. Management consulting still operates the same as in 2010, for one. ↩︎
  22. “Foreseeable future” is not centuries in the middle of ongoing singularity, but it’s the best we can plan for from an economic perspective. A favorable interpretation of Eliezer’s tweet certainly is “deep into the hypothetical AI superintelligence explosion when the cost of a core technology change will be negligible compared to the opportunity cost and the marginal improvements it brings.” ↩︎
  23. The wisest answer for product builders is usually a variant of “build my core differentiators, buy everything else.” ↩︎
  24. Noah Smith makes the case for unexpected market incentives around opportunity cost – wasting AGI on running many business processes will not make sense, since comparative value of existing human-driven processes will make it economical to invest our available compute into higher value opportunities for a very long time. ↩︎
  25. This is powerful in another way – your product is suddenly one of the devices to bring AGI about to the real world. Done well, that by itself may be one of the most impactful things we can do in our lifetime. ↩︎
  26. Gwern actually significantly disagrees with my conclusions here in an interesting way. In short, his argument is that (a) one should consider a genius-level programmer (consider: “digital Donald Knuth”) rather than a highschooler in the AGI thought experiment; (b) genius-level programmers consistently singlehandedly and speedily create and rewrite vastly complex software systems that would normally employ big teams of software engineers for years (so, are truly “100x”) and even big products and IT projects can be made technically trivial while “one person can hold the whole thing in their head”; and (c) Noah Smith’s opportunity cost hypothesis doesn’t hold well due to diminishing returns on intelligence – it won’t make sense to concentrate too many agents on a single task, creating a surplus that will go for those “regular peoples’ jobs” (including software engineers).

    And I can’t disagree! I think this scenario isn’t improbable either, but neither of the assumptions is certainly true on its own. (a) Geniuses on the Knuth or von Neumann level are extremely rare and their way of thinking may be much harder to replicate by AI; though, maybe you don’t actually need geniuses. (b) Software savants are easily hardcapped by even just gathering requirements and feedback on their work, and generally by how fast humans can understand and change any processes that involve people (or even other systems that the AI doesn’t have under its control). (c) Most of compute can still go into further AI training and exploring many parallel scenarios for key projects (1-10 von Neumanns could have built the atomic bomb if they knew which high level approach works, but maybe not run through the Manhattan project as a whole, which meant exploring the full hypothesis space including many blind alleys), then it’s not clear how many free agents there will really be to go around.

    Even if I think Gwern’s world is also plausible, ultimately such “faster AI takeoff” scenarios seem unactionable at this point – there’s no way to prepare for them, we can just enjoy the ride. Thus, I would rather spend time preparing for at least a somewhat probable scenario where my actions matter, rather than hypothesising all the scary scenarios I can’t do anything about. ↩︎

Related resources

Sign up to our newsletter

Ready to get started?

Make a quantum leap in your document processing approach. Boost accuracy and effectiveness with an AI-powered data capture solution for all documents.

Request Demo

Free Trial