How to Create a Generative AI Pilot Without Freaking Out Your Community

The ubiquity of new Generative AI tools transcends barriers of technical expertise and age. Even normally tech-averse family members are dabbling with AI. "At Passover this year, my 75-year-old aunt used ChatGPT to come up with a modern take on the ten plagues of Egypt," quipped Joe Lazer, A.Team’s Head of Marketing, at the Generative AI Salon on June 17 co-hosted by Baseten and A.Team in New York City.

But which of AI’s many uses will survive 2023’s hype and hoopla to prove truly valuable, sustainable, and profitable—and which will be remembered (no disrespect to Joe’s aunt here) only as seder-table novelties?

To help discern the signal from the raucous noise, A.Team and Baseten have begun hosting a series of monthly salons and hackathons to foster a community exchange about the burgeoning technology, and help technologists, executives, and builders discern the signal from the noise. If Generative AI promises a tsunami of disruption and growth, A.Team’s founder, Raphael Ouzan, noted, "We're in the eye of the storm. Let's be agile and nimble; accelerate into the wave."

At June's Salon, A.Team put three guest in the hot seat to share insights on how they’re “getting to value” with Generative AI. Mike Ritchie of Definite shared how his data analysis company is testing ways to replace complex dashboards with bots resembling friendly Slack-based data scientists. A.Team's product lead Matan-Paul Shetrit showcased internal experiments using AI to speed sales meeting prep and coach builders on their A.Team profiles. And Sid Shanker from Baseten revealed the pitfalls he’s seen as companies commercialize and scale their homegrown AI models.

Several key themes emerged on how to make AI-powered tools truly valuable:‍

Move Fast, Then Brake Things‍

Hackathons, where developers brainstorm and build products in a matter of days, have become vital in AI-powered product development. Even ChatGPT, OpenAI’s game-changing chatbot based on its GPT-3.5 and GPT-4 large language models, was the product of one such sprint. Shetrit explained how A.Team’s devs had promising results with two hackathon-built tools: one that analyzes sales calls to help salespeople tailor their the A.Team network’s offerings to match customers' needs, and the other to coach builders in A.Team’s network to optimize their online profiles. 

But he added that neither tool will launch widely until they validate that there are no problematic biases in the AI’s methods or results—a tough order with a large language model that analyzes countless variables. There are questions of both quality, and perceptions of quality: "How do we roll this out so it doesn't freak out your community?" asked Shetrit.

At Definite, Ritchie said, they’ve built a test system that inputs extreme cases to gauge the stability and consistency of the AI’s results. They’ve focused use cases on projects that have results that are verifiably correct or incorrect — for example, a concise SQL query that pulls data in a specific way — rather than more qualitative applications, like copywriting.

"Before diving in, you have to consider, what's your tolerance for error?" said Shanker.

Keep It Simple

Start with simple applications and ready-made tools before getting truly ambitious.

Given AI’s already dizzying complexity, limit the variables you supply a model, especially in new use cases. A "triage prompt" – a generalized ChatGPT prompt that helps break down a complicated process into manageable chunks—and that can then be fed to more specialized threads and prompts in ChatGPT or other AI’s, reducing the chance for error and misdirection.

Shanker described how he coaches clients to focus on dry, intermediary steps—something as simple as logging data usage and results, then implementing a feedback loop for quality assurance to greatly improve results.

Humans (Still) Needed

All the speakers noted how vital it was to keep human expertise in the AI loop. Sure, generative models often sound and look impressive, but they can be annoyingly superficial at times, not to mention plainly wrong.

Generative AI doesn't work like a magic wand, noted Sid Shanker of Baseten. In a recent pilot project with Patreon, the company combined AI models with human influence to provide closed captioning on videos in over 100 languages all over the site. This helped them create a feedback mechanism that helped them improve and fix transcriptions when they eventually included errors.

There are very, very few instances when you're going to be able to take a business problem, sprinkle GPT on it, and get results.

Neither AI nor humans could handle the task on their own. However, by designing a feedback mechanism that helped humans identify errors, fix transcriptions—and send that feedback back into the AI—put them on the path to completing the ambitious product. It’s not a matter of replacing humans with AI, but of getting them to work together.

After all, the “end user” is inevitably human in every value chain. "There are very, very few instances when you're going to be able to take a business problem, sprinkle GPT on it, and get results," Shanker said. "The key insights are going to come from understanding your domain and your customers."