Encouraging AI Experimentation Without Buying the Hype
Saying Yes Carefully
Presented by:
Joey Nutinsky, co-founder and CEO of Ruvna, shared how schools can encourage staff to experiment with AI without buying into the hype. He explained how AI chat tools work and argued that many AI products pitched to schools are a carefully written set of instructions on top of someone else’s model. He described how his company made trying AI easy: an AI account for every employee, a simple use policy, a channel for sharing wins, and lunch sessions where colleagues show what they’ve built. He showed how saved, reusable instructions can do work vendors charge for, and offered questions to ask a vendor before signing. Attendees asked about tools that monitor AI use, letting staff pick their own AI, and putting private data into AI tools. Technology leaders fielding a steady stream of AI pitches will find a practical way to decide what to try and what to keep.
Transcript
So welcome everyone. Today we've got an awesome topic.
We had a lot of people sign up, a lot of interest in this one.
We're going to be talking about how to say yes carefully to encourage this AI
implementation, but not just buying into all the hype.
So a really great session, a very timely session for all of us.
So we are joined by our friends over at Roofnet, and we have their CEO and
co-founder with us today. Joey, welcome.
We're so grateful to have you with us today.
Oh, thanks so much. I am so excited to be here, and my team has gotten pretty tired
of me ranting about AI for months and months, so I'm excited to have a new audience
to rant to.
I guess that was my cue to start. Hi, everyone.
It's nice to see
some familiar faces, some new faces. My name's Joey Natynski, I'm the CEO.
I'll do an intro in a second, but, again, thanks for joining and spending some time
with me to talk about this. So today's session is saying yes carefully.
It's about encouraging AI experimentation at your school without buying into the
hype. My goal is that you leave with a clear way to think about the AI products
that you're being pitched and a few things that you can try tomorrow, next week,
without buying anything new.
So a little about me. I co-founded Roofnet in college with two friends, but my
background is in software development and product design, and even still now as
CEO, I still spend quite a bit of my time working with our product and our
engineering teams. So when I look at an AI product, I tend to think about how it's
built and not just what the demo shows.
Roofnet builds safety and operations software for independent schools, so things
like emergency accountability, visitor management, attendance, and new this year is
dismissal. That's the last I'm going to pitch our product.
But I want to be clear that we are not an AI vendor.
We're very deliberate about what goes into our product and what doesn't, AI
included. So when the question comes up, we start with what problem would this
actually solve for a school? So this puts me on both sides of the AI conversation
that I think many of you are probably having day in and day out.
As a vendor, I get asked, "Are you adding AI?
Can we use AI to do this and that?" And as a buyer, I get pitched AI tools just
like everyone else. And on our team, I'm the one who evaluates the AI tools for our
team and manages what we use, what we pay for, and what we say no to.
So everything that I'll be sharing today is going to come from that lens.
Any questions that you might have, feel free to either send them in the chat, you
can speak up, and then I'll also have some time at the end that we can discuss
questions.
So here's the idea that I want you to hold onto for the next hour.
Trying an AI tool and keeping one are two very different decisions.
When you treat them as one, it's easy to end up saying no to both.
So keep this in mind, make trying cheap and make keeping a decision.
Many schools aren't yet at a place where trying is cheap.
Maybe that's because AI is blocked or because nobody has sanctioned a place to try
it. So the first half of the session is about opening that door and encouraging AI
experimentation. The second half is about deciding what's worth keeping once you
have opened that door. Before we get there, let's start with a quick review of
what's actually happening under the hood.
Most of you use AI tools already, so probably none of this will be news, but I find
that it actually helps to remind myself this very often because it's very easy to
forget once these tools start feeling like magic.
At its core, every large language model, so whether that's ChatGPT or Claude or
Gemini, is a word-guessing machine.
You give it words, and it predicts the words most likely to come next.
Then it does it again and again until it has an answer.
Here's how I like to think about that full loop.
Step one is the words that go in, and it's usually a few different pieces combined.
So there's your request, which is the part that you typed into the chatbot.
There's context, like a document you attached or data pulled from another system,
and there's the whole conversation history so far.
Keep in mind, the model doesn't remember you at all between messages.
So we call that being stateless. Every time you hit send in that chatbot, the app
is quietly sending the entire chat history back in, and so that makes it look like
the model's picking back up where you left off.
And that's why, for example, long conversations can start to drift, and eat up
context faster because, again, it's got that whole history, and why starting a
fresh chat sometimes fixes things.
It's also why as soon as your model does something you don't like, instead of
trying to steer it, you want to start a new conversation.
Because if you're trying to steer it, it still has the incorrect steps that it took
as part of the input it's getting on every new output.
So at the very top of that input, there's the system prompt.
It's the set of instructions written ahead of time that you usually never see.
It tells the model who to be, what to focus on, what to avoid, and what format to
answer in. It's actually
arguably where most of the magic happens.
The same model can act like a writing coach, a tutor, a policy reviewer.
The difference is mostly in the system prompt.
Coming from a programming background, this is the piece I think that surprised me
perhaps the most. I'm used to code, being what drives behavior.
But in large language models, it's just plain text.
If you want your chatbot to be named Siri or Claude, you'll notice the first line
in those system prompts is, "You are Claude, a helpful chatbot."
So step two,
the model runs and does its word guessing.
That's matrix multiplication and certainly out of scope for this.
But that's the magic black box. Step three, words come out.
So that could be a summary, a draft, or a plan, but it also could be a request to
use a tool, like get the weather for today.
Again, the only thing that the large language model can output is text.
So when it wants to do things that are smart, it's actually part of step four, code
that is listening to the output of the large language model and saying, "Oh, the
model now wants the weather," or wants this student's schedule, or wants my
contact list, whatever it is, and then quietly does a new chat loop in the
background without you even noticing.
So keep in mind, again, the model never touches your systems directly.
It writes words, and then our own code that we write acts on them.
Here's why this matters.
Most AI products right now are what I refer to as wrappers.
Only a handful of companies are actually building the actual models, and everyone
else takes one of those models and writes a really careful, good system prompt,
puts an interface on top, and sells it.
Why pay no attention to the man behind the curtain?
In a lot of the AI pitches that you're getting, the man behind the curtain is a
fancy system prompt and someone else's model that they're renting.
That's not a criticism. A great system prompt takes real work.
I know that firsthand, and some wrappers are absolutely worth paying for.
But once you know where the value lives, you get to ask better questions, and often
you can even try to write that prompt yourself. Let's look at an example.
I extracted this yesterday, maybe two days ago, off of my Mac.
This is the actual prompt. So
for anyone using a Mac or an iPhone with Siri, AI, and writing tools, you can
highlight text and say, "Rewrite this to be more professional." Under the hood,
when you click that Make This More Professional button, Apple is sending the text
that you see to the right into its own model.
In Apple's case with writing tools, that happens to be a very tiny model that's
running on the phone usually.
But the value here that they're adding in addition, again, this is a special case
where they happen to run the model, but all the AI tools you're using really work
like this under the hood. It's that polish button in the menu.
It's the system prompt. That's the value that's being added here.
Raises a fair question. So if I could paste my own paragraph into ChatGPT and write
make this more professional, what's Apple really giving me here?
Like I said, so this is a button right where I'm already working, so I don't have
to switch apps. It's a prompt that they've already written and iterated on and
tested and tuned so that it does the job well.
And it's a model that they chose for the job.
Plus, again, the plumbing that takes the result and swaps it in for my original
text.
Those are totally real conveniences.
And for a free feature built into my laptop, it's great.
But notice the part that's doing the actual thinking is the same thing that you
have access to, a model and a prompt.
So every AI product like this breaks down into the same constituent parts.
There's a harness, right? And that's the interface, the plumbing, sometimes the
integrations with other systems, which the way that those integrations work is at
the top of the system prompt, is a list of you have access to this tool, and this
tool, and this tool, and please ask in your output if you'd like to use it.
So,
integrations with other systems, the code that assembles the prompt and handles
what comes back. For writing tools in that example, it's the menu button in the
Apple Intelligence interface.
There are the instructions. Those are the system prompt that the vendor wrote.
In writing tools, it's literally just rewrite this in a professional tone.
Then there's your content. So that's whatever you selected or uploaded or
connected. Here, it would be that highlighted paragraph.
And then there's the model. Like I said, most companies are renting it from one of
the few AI labs and running A/B tests to see how cheap of a model they can run
before the user notices the output slips. So keep all of these things in mind.
We're going to come back to who owns each one in a bit.
So let's apply that to the kinds of products that are probably landing in your
inbox.
First, let's take an essay feedback tool.
Behind the curtain, the system prompt probably says something like, "Act as a
teacher and give constructive feedback instead of just fixing mistakes.
Refuse if the student tries to get you to write the whole essay for them." And then
it's going to attach the assignment details, the grading rubric, and the student's
essay. And it'll probably include instructions on how to format the answers that
the app knows, like how to take that output text and stick it in the interface.
Let's look at maybe a lesson plan generator.
That prompt is going to have the lesson plan template, the standards set by either
your state or your school or the teacher or a mix,
and
how to write the output. Again, I'm not saying these products are bad, and I'm
certainly not saying they're easy to build well.
But when you look at them this way, the core is something that you can understand,
and in a lot of cases, try yourself.
Hang on to the lesson plan generator because we're going to come back to that as an
example later.
So let's take a quick moment here. Someone, if you want to either type in the
chat or if anyone wants to speak up,
does anyone have other good examples of AI tools that exist really as wrappers
around an underlying model?
Flint AI is a good one for K12.
Yeah. And, can you talk a little bit about what that does?
I think its key thing is
making alerts to teachers if the tool is being used in unsafe ways,
making sure to always use the Socratic method-
Mm-hmm
... and keeping kids from skipping critical thinking.
Mm-hmm. Yeah, and so you can imagine all of those different features exist
somewhere in their code base as a markdown file, right, of the instructions, of the
text that's going to go in and tell the model what to do in a given scenario.
Like,
if this input does not contain proper Socratic method ideas, make
sure you respond and tell the student to change X, Y, or Z, right? Any others?
All right, let's keep going then.
So what's the problem here? Everything I've discussed is
totally fine, right?
The problem is that, in my opinion, is the market that all of these things are
being sold in. There are lots of forces at work right now in this market.
So the first, we're all in growth mode.
The AI labs are spending unspeakable sums of money on computer power and pricing
their models to win customers, artificially lowering those prices.
What you pay today reflects a race for market share more than what it actually
costs to run.
All of the large AI vendors are losing huge sums of money on actually running their
models. Second, there's constant change.
Everything we're talking about today has the potential to be out of date in three
weeks, if not sooner. New models, new features come out every few weeks, older
models get retired, and that happens on the provider schedule, not yours.
Third, and I think this is a really big one, is that nobody's found the winners
yet. The use cases that will matter long term and the interface paradigms that will
matter long term haven't been figured out.
So everyone from two-person startups to the vendors that you already use are all
shipping AI features as quickly as they can to find out what's going to stick and
to avoid being the one left behind.
And fourth, it's never been cheaper and it will never be cheaper to launch these
sort of products and features. Put a subsidized model together with a good system
prompt and you have a product in a few weeks.
So that's why your inbox is full of these AI pitches.
Plenty of these products probably won't exist in a year.
So what's going to happen when the market actually figures itself out?
The bubble bursts, as some of us refer to it as.
There will be a couple of consequences.
One is we'll start to see swapping in cheaper models at the same price, which
likely results in quietly decreasing
quality. For those of you that use ChatGPT, there's a really good example there,
which is over the past few months,
users have lost the ability to actually choose which model is answering their
prompt. So it used to be that I could choose,
first they were just numbers, like GPT-3.5, 4.5, and then they have the buckets of
models like Astra and Sol and whatever.
Now it's just default or please try extra hard.
And the reason that OpenAI does that is because they don't want the user always be
choosing the most expensive models to run.
They want to use their own intelligence to see how cost efficient can we make
running these features before the users notice.
Prices will raise, usually right after your staff have built their workloads around
whatever the system is. A lot of these tools are
likely to just stop existing or go out of business.
Or they'll get absorbed into tools that you already pay for.
And this one is certainly already happening.
A year ago, so many things that were being sold as independent standalone products
are now just built into whatever main AI provider that you're using.
And we'll certainly see more of that because just like what I'm encouraging you to
do here, the big AI providers are trying to see what use cases are going to stick
and provide value.
So to summarize,
is AI early, full of fancy wrappers around just a small number of models?
Yes, absolutely. But can you afford to sit it out and wait for the dust to settle?
No, absolutely not. This technology is immensely powerful and it isn't going away.
The tools will change, but the way that our students and our staff will work is
already changing with them. And independent schools don't sit things out.
Trying new approaches before everyone else is part of what makes our schools what
they are, and families choose us partly for that. AI shouldn't be the exception.
And third, your staff are already using these tools with or without a plan.
Blocking it mostly pushes that use into personal accounts that you can't see with
your school's data going in there, too.
So the real question isn't whether AI gets used at your school, it's where it
happens.
Here's how I think about it.
On one side, the market often isn't ready to buy from yet.
Adding any vendor is a long-term commitment, a contract, data agreement, training,
renewals. With AI, prices, models, and the vendors themselves all still shifting,
and like I keep saying, many of these products will look very different a year from
now. But on the other side, every pitch you get is a free idea.
Those vendors are spending their own money to find out which use cases work.
And when we know that most of what they're selling is a prompt and a model, you can
prototype so much of this yourself on the AI that you already have and let you
and your staff learn a ton along the way
Build your own while the market matures.
And when dust settles, you'll know what you actually need from a vendor, and
whether you need it at all.
This is what I mean by making trying cheap at your school. How do we do that?
First, we need to give staff a place to experiment.
So that means a school-managed AI workspace, so that experimenting happens under
your data terms and not in personal accounts that you can't see.
At Ruvna, we've standardized on Claude as our main AI platform, and every employee
gets an account on our team plan. Nobody has to ask, and nobody has to expense a
personal subscription.
That one decision did far more for adoption than anything else we've done because
it took away the question of whether people were allowed to try.
And speaking of subsidized prices, we're in this
really
special moment where we get to pay for subscriptions to AI vendors
that give us much more access than if we were actually being billed at API rates.
Right? So I think a Claude Team subscription's like $30 a month, but for all the
Claude usage that we're actually using, that costs Claude on Anthropic way
more than the $30 a month. That's not always going to be the case, but for right
now, those subscriptions are really, really great value.
Now, the good news is that you probably already have one.
Google includes the Gemini app and Gemini Notebook for free in its Workspace.
They change what's in Workspace like every couple of weeks, but last I checked, the
Workspace for Education editions had that stuff included.
Plus, there's a paid upgrade to put Gemini in Docs and Gmail.
And I'm sure Microsoft-- I'm not as familiar with the Microsoft stack, but
they probably have about a million and a half Copilot branded things as well.
So like I said, buying into an AI workspace and an AI platform for your team,
for your school, gives staff a place to experiment and to try.
Second, write a policy. Make sure that policy is saying yes while still upholding
your standards. The encouraging half is simple. Right?
Use anything in the school workspace freely and share what works with your
colleagues. You want people experimenting where you can see it and where others can
see it too and build on it and innovate.
The other half is where you need to hold the line.
Things like no student, family, or personnel data in personal accounts, AI-assisted
work needs to meet the same bar as any other work, et cetera.
On my team, on our team,
it is totally okay to use AI for deliverables that we're sharing with each other
in meetings, but the quality bar stays exactly the same.
If someone hands me AI slop, it's not the same as actually handing a
polished, ready deliverable. So the goal is use, more use in fact, without more
slop.
A good rule of thumb when you're writing these policies is if you wouldn't put your
name on it, don't send it. Right? Whoever hits send owns it, whether it was AI
generated or not, they are responsible for the output that it's producing.
Third,
think about who's doing the trying.
In every industry, not just schools, lots of people feel like AI is being forced on
them.
Mandates,
management saying, "We need to see AI numbers go up," et cetera.
Now, mandates like that usually get you resentment and people checking a box rather
than actual good faith experimentation and buy-in.
But we do have people who are eager to dive in, and it's those people
that we want to invest the tools, the time, and the permissions to prototype.
One way to do that that I've seen is making an AI working group.
Maybe it's a mix of teachers and admins whose job it is to build these things and
share what works and report back.
It's important framing. It's a group that is building and not another approval
committee. So the people that are willing to and interested in getting their hands
dirty, that's worth giving them some school budget to play around in and innovate
and see what's possible. And then whatever they're successful with and where
there's real value, we'll look at how that works in a bit.
That is what we can look at scaling up.
What do I mean by building and prototyping?
One of the best places to start is skills.
A skill is a set of instructions you write plus any reference files it needs, and
it's saved in your school's AI workspace.
So skills are a relatively recent concept.
Before skills,
I personally used to have
many, many, many docs and files full of saved prompts that I would use for common
use cases. Right? I'd work really hard on a prompt and want to remember it and want
to iterate on it and get better. So I would save maybe a big Google Doc with like,
"Here, use this prompt in this case, use this prompt in a different case." Skills
is essentially the same thing, but a thousand times better.
So a skill
is a collection of-- It's essentially a prompt that gets to be triggered either
when the model decides or when you invoke it.
We'll look at some examples here in a second.
But it takes the manual work and distribution out of those saved prompts.
So when a teacher, for example, figures out a great way to do something, a skill
turns that into something that the whole team or department can use.
So let's look at a real example that we use at Ruvna
at least once a week. So I have a weekly leadership team meeting with the
senior company leaders.
Every week, maybe a day or two before that meeting happens, I run this skill to
prepare the agenda and identify all the open items. Here's what that skill does.
We'll look at the actual skill in a second, but for now, just a summary.
It goes through the recorded meeting transcripts from our past three or four
leadership meetings. It goes through the agendas and notes from those past
meetings, and it's connected to Slack, which is what we use for internal messaging,
and goes through our channels to figure out things that have been already handled.
I saw a question out the corner of my eye.
I'll get to that. Macy, I will get to that in a second. It's a great question.
So what the skill is able to do is figure out, okay, this was listed last week as a
follow-up item, but I can see there was a Slack message that identified that it was
already taken care of. So it does all that and then comes out with our standard
recurring meeting items, the open items from previous meetings,
what the next step was, the exact moment that they were discussed in the previous
meetings.
I see there are questions about what I mean by skills.
I am referring to
agent skills. So this is an open protocol.
Anthropic was the one who originally created the framework of skills,
but it's been adopted by the other AI vendors.
So I think Gemini used to call them gems.
Now they've adopted the name skills as well.
It's literally a zip file
that inside of which is a markdown file named skill.md, and markdown is just
text. So,
a great way to write a skill, ask Claude to do it for you.
In fact, ask Claude to go through your recent Claude usage and identify what skills
would speed up your workflow. And if you're in a team workspace, you get to
share these skills with each other.
Here's the actual filtered with some irrelevant details removed,
but here's actually what the skill is.
Here's the prompt that every time I invoke this skill, this auto message is sent
into the model. It tells it what to do, what sources to look at, the steps to take,
certain rules to follow. It's plain English.
We've tweaked it over time, and the initial draft, in fact, was written itself by
Claude.
Skills, I cannot overemphasize how valuable these are and how many--
A use case like this of creating a smart agenda based on your recent meetings,
there are hundreds of vendors that are trying to sell me, Ruvna, products to do
just that. And literally, we can do it all with a markdown file and the AI
subscription that we already have.
A really good example of a skill that you could write would be to write in your
school's voice. So you would be able to write this skill that explains
the sort of terminology that our school uses.
For example, we never refer to it as high school.
We refer to it as the upper school. We refer to our
families as families or as members or whatever that is.
And any guardrails about make sure it never says this or that.
And then anytime anyone on your team is working on something that also needs to be
phrased for public consumption, so maybe I'm in a conversation about drafting
a letter home about something that's going on, this skill can get invoked, which
just injects that prompt into my conversation, and all of that saved nuance gets
applied.
Here's some more skill ideas. Claude came up with a lot of these with me.
So one is like-- I'm not going to go through all of these, but a couple that I
thought were really interesting.
On our end, like in Ruvna, we have found
analyzing our help desk, our Ruvna support, with AI and understanding what the
common themes are has helped us tremendously to identify patterns,
to
create how-to articles from common issues.
EdTech vendor policy reviews is another great one, right?
So every vendor that you vet, you probably go through a pretty similar
set of steps for. And with a skill, you can package that up and most importantly,
iterate on it and improve it over time,
to go through the privacy policy and check does it meet this standard, that
standard, and produce for you a tailored list of five, 10 recommendations.
So there's an important part with everything I'm talking about today of using your
own AI stack to sort of quick and dirty prototype your own version of these
products that you're being pitched, which is that a lot of them require access to
some kind of data.
The Model Context Protocol, MCP, which is another standard Anthropic
created, is the standard way for
AI to integrate with third-party systems.
It's essentially similar to what an API is, but instead of it being defined by
endpoints and parameters, they're defined in plain text.
It says to
the model, "You can list attendees in a Zoom by asking me to list attendees in the
Zoom." Right? And then when the model output says, "List the attendees in the
Zoom," some code captures that, grabs the attendees, and sends it back into the
model. So that's the root Model Context Protocol.
A growing number of vendors publish one. That list will continue to grow.
So if your vendor has an MCP available, then your own AI and your own
skills are able to connect and either read or read and write to that data directly.
So in our case, with the
leadership meeting agenda skill, it's connecting to our internal wiki at
Confluence, it's connecting to Slack, and it's connecting to where all of our
meeting recordings live.
Not every vendor has an MCP connection.
So option two there would be is the data already there on your computer?
If you can download that data as a spreadsheet once it's on your computer, then the
model is able-- Well, the harness is able to take that file and give it into the
model, right? So it could be like, look at this spreadsheet and do A, B, C, or D.
Three would be that you still write the skill, right?
So let's say neither option one nor two are available.
Oh, sorry. I got distracted here. Never mind. Okay.
So if the vendor doesn't have an MCP connection, if you can't get the data on your
computer, you can still move forward with writing the skill and standardizing what
the prompt is, what the instructions are, and separate out the concept of these
instructions and the data. And the instructions can still get written, and
then you can hand it the data that it needs when you want to actually go and run
it. And option four requires a little bit of bravery, but is you can build your
own. In fact, in our AI users so far, we've built a handful of custom MCP
servers and custom tools that
connect to a system that doesn't provide an MCP or that
provides access to some internal resource or system.
And we have Claude write those MCP servers itself, right?
So the barrier to entry here is truly, truly accessible.
Now, once people are trying things, it's so important to make those wins visible.
A policy makes AI use allowed, but seeing a colleague's win is usually what gets
people to actually try it. At Ruvna, we have a Slack channel called AI Wins.
Actually, I checked that was wrong. It's AI at Ruvna. Same thing.
Anyone can post a small anything that AI helped them do.
I identified that this prospect
talked to us six weeks ago, anything like that.
When
people see their coworker was able to save 20 minutes on something that they also
hate doing, that's what's going to get them to try it themselves.
At Ruvna, we also run AI lunch and learns.
They rotate among staff, but each time someone shows how they're using AI, what
they've built, and the tips that they've picked up.
And it's colleagues showing colleagues, which carries quite a bit more weight than
a training session from IT, I'm sure as we all know.
As I mentioned, at Ruvna, we have one AI platform for everyone.
We happen to use Claude.
The lunch and learns have an owner.
They happen consistently, don't depend on anyone volunteering.
We use shared skills for recurring work, like the leadership agenda that you saw
earlier, and any employee is able to donate
their skill to the team, so it doesn't take elevated permissions to share a
skill with everyone else.
Every so often, I pull our usage numbers together, and I saw there were some
messages about this
in the chat. So every once in a while, I pull our usage numbers together, and I
share them with the whole team.
It's nothing fancy, right? But it's just a cool way for everyone to see who and how
we're all actually using AI. So the chart on the right here is from one of these
over the summer, and here what I was looking at is how many days in the month each
person on our team used Claude at least once.
You can see it's close to a good bell curve there.
A couple people are in it every day.
Most of the team lands in the middle, and a few are still just getting started.
And finally, anyone can request a new AI tool.
Like I said, the space is changing so quickly.
It's easy once you start making policies and having systems to sort of close out
and put blinders on to what the rest of the market is doing.
To avoid that, we still encourage our employees to sign up for a free trial of a
given vendor, to make a pitch to us about having us buy them a subscription for X
number of months to try something out. The approval rules depend on the cost.
So if it's a $20 a month subscription,
absolutely, we'll let the employee try it for a couple of months and see if they
get value. And then if so, we continue this loop, which we'll talk about in a
minute, of how to expand that access.
So this is the same idea as the whole talk.
Trying things needs to be really easy,
and keeping things is where the decision comes in.
I see a good question here about what are the biggest differences between Claude
and ChatGPT and Gemini.
The answer to that is not very much, and it changes every couple of weeks.
This is a crazy time in
history because new, what we call frontier models, are coming out every
couple of weeks. So one day,
Anthropic will release a new Claude model that's better than anything the world has
seen before, and it's like, "Oh, Claude is way ahead of the other two players for
the next couple weeks." And then OpenAI will leapfrog them and come out with a new
model. And then Gemini will leapfrog them and come out with a new model.
So they're all sort of converging around some standards, like I mentioned, with
model context protocol, with skills.
And the nice thing about owning your own instructions, these skills, and these
prompts, is that you can switch the underlying model very easily.
They all just read Markdown. So my skill, if I decide to switch from Claude to
OpenAI, I'm not locked in. We can just move everything over very simply.
In fact, some of the internal tools that we use--
Well, I don't want to spoil that. We'll get to it in a minute.
So, so far we've been talking about how to make trying easy.
But eventually, some of these prototypes are hopefully going to stick, and people
will start depending on them. And of course, the market will, again, hopefully
mature. Some products are going to earn a space in your school.
When that happens, that's where we need to decide where does this thing live.
So here's how I think about the options.
We're going to look at the lesson plan generator that we talked about briefly
earlier, and four different ways that that could be implemented.
So starting on the very left, prototyping on your own AI stack could mean skills
and connectors and shared context on the AI that you already pay for.
You write that prompt, your AI provider supplies the model and runs the app, your
data stays under whatever data agreement you already have.
Most ideas, most of the time, should start here.
Next, if you're feeling brave or you have someone on your team that's technical,
you can build on the provider's API.
At that point, now you own the prompt, the app, and exactly what data gets sent
where. That used to require a developer, but today, AI can actually do a lot of
that building for you. You'll see one that we built here in a minute.
Another option, one that we're seeing some adoption with, is called bring your own
key. Right? And that's where
you have the account with the AI provider.
So like I
as the user,
have the account with OpenAI, and I create an API key, and I give that API key to
whatever tool I'm using. So in that case,
the tool is still the thing that's running the app and writing the prompt, but
you're plugging in your own model account.
So that means that the model's pricing and data terms sit under your contract, not
theirs. Unfortunately, what you don't get there
is control or ownership of the prompt.
And then all the way on the right, you can see you could go to a full-on vendor
product. At that point, they own everything, including the model that's underneath,
and of course, that can change without you knowing.
Every step you're moving to the right on this chart, you're owning less and less,
but you're also maintaining less and less. So neither is wrong.
The question is, what are you getting and what are you giving up?
I was wrong earlier when I said we were looking at the lesson plan generator.
Now we're looking at the lesson plan generator.
So if we wanted to prototype a lesson plan generator on our own AI stack, that
could be a shared scale in a template that says, "Here's what lesson plans in my
school look like, and here are a couple examples of really good lesson plans."
A department could build that in less than a week and start playing with it and
seeing and really evaluating, does this offer us value? Is this saving us time?
Is it producing better lesson plans than we were writing already?
You could build it on the API, right?
So it could read your curriculum map and say brass straight into your LMS for
teachers to review, for example. That'd be a step up.
You'd be able to reuse most of the prompt that you wrote in step one of prototyping
on your own stack.
A full vendor would start to make sense when it brings something that you can't
really build yourself. So whether that's a really deep workflow, a curriculum
library, et cetera. So for most schools today, I would say the prototype wins.
Revisit the other options as the market matures and as you learn what you actually
need.
To be fair here, especially as someone who sells software to schools,
wrappers can be worth paying for. We pay for some.
They maintain the integrations with your system, so you don't have to.
They can hold your data that you can't easily rebuild.
They can take on the risk and maintenance and liability, so it isn't just on you.
They can make workflows possible that you otherwise wouldn't be able to build.
It's not that being a wrapper is a problem. It's not.
The question is whether you're paying for something real or just double paying for
the things you're already paying for in your Gemini and Claude and OpenAI
subscriptions.
Here are some examples of things that we said no to.
Zendesk, which is what we use for all of our customer support,
offers a range of AI add-on that we've evaluated.
They change every couple weeks because, again, the field's changing so quickly.
And we said no because we
already have an AI provider, and we built our own little Zendesk MCP connector
and we built our own, essentially,
chatbot that's able to connect our support tickets.
So as the support specialist answering a ticket, I'm able to open up this internal
tool, which is using our own API keys, right?
So it's billed straight to Rootin and it's stored in some Postgres database on our
own stack. And it's able to pull in the ticket, it's able to connect into our
knowledge base, and it's able to connect into our internal systems, and it
understands what we care about when writing support replies.
And it's also intentional that we don't have it directly integrated into our
customer support system because I don't think any of us like receiving AI-generated
output when we're trying to reach customer support, right?
The way that I built this system that I'm talking about here, I call it Support
Bot, and for me, knowing the answer to a given support issue was never the hard
part. Phrasing it like a human, for me, has always been the hard part, right?
So what we wrote for Support Bot was, how can I take an explanation like I'm
talking to a colleague of, oh, this school's issue with that they need to do X, Y,
and Z, and turn that into something
that we could actually share with a customer.
But then it takes a couple tries, right?
Iterations, and still it's the human that's putting it back into our customer
support system and hitting send.
Same thing with Google Workspace's AI add-on.
We already pay for Claude, like I said, and that can connect to Gmail, into
Calendar, into Drive. So in those cases, we didn't feel that it was worth it, and
we felt that particularly these vendors that are adding AI add-ons onto their
existing tools, they're trying to stay relevant.
They certainly need to prove to shareholders that they're not missing out on the AI
wave. But when we look at what value they're able to offer us compared to what
we're already paying for, plus a little bit of elbow grease in writing the prompt,
those two didn't make the cut for us.
Here are some questions that I think could be valuable to ask your potential AI
vendors when you're in meetings with them.
You want to say all these as respectfully as possible, right?
But
what can you do that a shared scale in our workspace can't, right?
What is the magic special sauce here?
And can our own AI tools reach our data in your product?
Those are the potential moats that you need to look for.
So with that,
like I said, let's bring it all together.
Most AI products right now are wrappers in an unsettled, hectic, crazy market.
And instead of rushing to buy, let those pitches inspire your prototypes that you
build on the AI that you already have, and save the vendor commitment for when a
product clearly, truly earns it. Make trying cheap and make keeping a decision.
With that, I'd love to take any questions.
You all feel free to come off mute if you would like to ask questions, and we can
also look through the chat now. There's been lots of great conversation.
And Joey, fantastic job, by the way. That was an excellent presentation.
But yeah, you all feel free to unmute if you have questions.
One thing I'm noticing in the chat is, how do we evaluate provider A versus
provider B? And something that we do on our team is, like I said, everyone gets a
Claude subscription, but we also have paid subscriptions for
ChatGPT and Gemini, but those are limited.
Like myself and a few other people in our core AI decision-making
group get access to that, because we don't want to be siloed.
And when you do have that access to all of them, you'll quickly see that they're
all selling you the same product, right?
The model itself is different under the hood.
So, for a while, Claude Opus
4.5 was really verbose and wordy, and then there were other issues with different
models. But the feature sets, they might have different names, but essentially
they're all the same.
Oh, there's a really great question in the chat.
How do you prevent scope creep within Roofnode?
So they were saying that they feel like with AI, a lot of people start augmenting
what they do,
what their software purpose is, and what they're trying to solve.
Yeah. We have not integrated any AI into our product.
I can think of things that would demo really, really well,
right? Like we have
an announcements product for mass notifications, right?
So like, "School's closed today," or, "Picture day is tomorrow." And could we
integrate a model in there that is helping suggest what you're writing for you?
Sure. But is that what the school's going to be paying us for? No.
With all of our product design decisions and with scope creep, we really try to
take a long-term outlook, and we try to avoid the temptation of
short-term marketing hype wins.
It takes a lot of practice, and we don't always get it right, but I think over the
long term, it's been successful for us.
So more questions from the chat. Jesse is asking the group if you guys have any
suggestions on allowing users to use their preferred AI, but basically having some
management controls, a management profile that restricts how it can interact with
the org's data. Jesse, before the group answers, when you're looking at the
enterprise level license, so for example, if you have the Gemini under your
school's suite of products, you're going to get some more of that protection when
you're signing onto those. Claude just released something for schools as well.
But yeah, group, you all help her out. Help them out there.
How are we handling that? Any input?
So
I have a kind of
question on this. There's this idea of like,
or I guess it's more of a concept right now, I haven't seen it actually work in
practice, of these network level products that monitor API calls with
AI products and data going into AI products.
I think one of the products is called Harmonic or something like that.
And so I would imagine if these products are successful, which I don't know that
they are, and this isn't something that we've actually explored yet, but having
some type of system level product that can monitor AI usage.
Although I think it's an emerging space, and I'm wondering if anyone else has
done anything with that.
I think that's a great example of
that-- I'm not familiar with that product, so everything I say here could be
totally wrong and is my opinion, not Rootin's, whatever disclaimers I'm supposed to
make.
I don't control what OpenAI does to their API or what Anthropic does to their API.
And so
the space is changing very, very quickly, and I would be very hesitant to pay for a
product that purports to be able to monitor something that they don't control and
don't know the future direction of.
I believe for sure that such products are being pitched.
That sort of thing raises my
BS detector a little bit of like, long term, is this really going to
do what I think it does?
To the original question,
I'm interested in
if you were to do a pro/con list of a let my staff use their own AI provider
versus provide for my school, if you did that pro/con already and you came to the
decision that let my teachers use whatever system they prefer is best, I'd be
curious how that worked out. When we did that evaluation,
choosing a vendor, even if it was the wrong one, we debated should it be Claude,
should it be OpenAI? And one of the things we thought about is like, well, what if
we choose the one that ends up losing the race?
Even if that's the case, one, switching isn't that hard, but two, you get all of
those enterprise controls when everyone's under your account.
And
within Claude, we get to control specifically which connectors are allowed to be
used, specifically which
specific security settings are allowed to be turned on and off, and it goes on and
on, and it changes all the time. So
when we're in such an emerging market like this, choosing a vendor and providing
that to your staff is probably the best way to exercise control.
But again, if you've done that cost-benefit analysis and there's a benefit I'm not
thinking of, I'd love to hear it.
To Jonathan's points as well, so I've kind of done a little bit of looking
into technology like that, where it's like AID or AI data
retention tools. And so for example, CrowdStrike, I think has something
similar to that.
I think where our concern is, and I think I understand the point where
a lot of these tools are emerging right now, and
it's hard to kind of-- you don't want to pick the loser in this race here.
Where my concern is it's like if you are putting in information into a
company's AI or a different vendor's AI, and they end up being the loser or they
get bought out by a different company, I think that's where the concern is where,
is that new company that purchased that company respectful of the previous
company's terms and conditions with your data as well?
So I think that's where I'm approaching it where
right now I have something in place, or we do have a tool that kind of monitors how
much or different AI tools that are being used, but it's not really controlling
what can or cannot be put into those LLMs at the moment.
Good point.
There's another question in the chat talking about stack creep and how to balance
the freedom to explore with the need to support and monitor product use
effectively. I think this is a great question, too.
I can tell you just by talking with a lot of other independent schools, the way
that many schools are approaching this is a mass rollout of whatever they're on.
So this year, a lot of people, again, if they were a Google school, they went to
Gemini, if they're Microsoft, they went with Copilot in that direction, as far as
the mass faculty rollouts, and then they would have a smaller set, like subset of
people, that are then going and doing more like R&D, poking around and
kind of their super users, but they're still given some guardrails.
But again, most of your users may not be able to max out the full capacity of some
of these pricier models.
Again, just something that we've observed, but I open that up to the floor for you
guys if you have any other input on that question, too.
It also sort of speaks to the thesis we've been talking about for the past hour,
which is
if you buy in well to one of these major providers, be it Google or
OpenAI or Anthropic, a lot of these other point solutions become unnecessary,
right? And a lot of them can be implemented with
plain text files on top of
the system that you're already deploying.
Scope creep is a huge benefit of doing something like that.
Not scope creep, stack creep.
Is stack creep a real thing? I just said it. I was just thinking.
I've never heard it before, but I love it. It's my favorite phrase of the day.
I think it's a good one for now, just because there's a part of us, especially as
educators, we want to provide teachers with the tools they need.
And if they go to a training and see something amazing, we want them to be able to
come back and try that thing. And for my team on the back end, it's making sure
that, like a cursory look to make sure it's safe and keep it in the yellow zone
where don't put any PI in, but absolutely.
And then if 10 people love it, then we'll start to consider it.
I think the thing that's different about AI tools is that could actually be the
tool next year or in two years. And we're talking about
whiplash for faculty, but
if there is something that's a better fit for us right now, we're heavy into Flint.
And if it turns out that Schmidt is better in two years, I want to switch to
Schmidt.
But keeping track of it and then making sure that everybody has it and then having
my team dedicate cycles to being able to maintain it, keep it safe, and then of
course pay for it. Those are the question marks I've got there.
Joey, another question from Scott, and he's asking, can you speak to the risk of
entering PDI data into--
entering that kind of data with vendors that provide enterprise data-level privacy
agreements?
Yeah. So all of the major providers offer some form of,
one of them calls it zero data retention, multiple of them.
They might all call it zero data retention.
So it's going to depend on your
specific enterprise agreement.
That said, it's also a risk and reward
thing like many of the decisions that we have to make are.
There are
consequences to sitting out the race, right?
And seeing what the other schools in your area are able to do with AI.
And there are consequences to being too liberal with your data and giving it to
folks that shouldn't have that access. And it's not binary, right?
It's going to be a spectrum.
I think that,
personally,
it's a point in favor of OpenAI and Anthropic to me over Google because they are in
the business of keeping that trust and providing the model inference.
And Google,
AI is not their business. Primarily, maybe on the earnings call, they're changing
that tune. And what they are in the business of is data.
And so, right, there's a good counterargument to be made that Google's been around
way longer and has earned more trust, right?
These are the judgment calls that need to be made.
But we're all speculating. So,
again, my BS meter goes up anytime someone is trying to definitively tell me, "Here
are the five golden rules to keep in mind with securing your enterprise data in an
AI world." Right? Anyone who's saying that is probably making it up.
Awesome. Well, y'all, we are at the top of the hour. That went by really quickly.
I know that there are still more conversations that we will continue to have.
We would love to see all of you at the Atlas annual conference in person to talk
about these there together, but we'll have lots of virtual programming in the
meantime. We appreciate Revna being with us today, and Joey, thank you so much.
You did an awesome job.
Thanks so much. It was great being with all of you.
And you should check out Revna if your school's not already using it for crisis
management,
student attendance, dismissal, emergency accountability, and school-wide
announcements.