Demystifying Data: Institutional Research at Independent Schools

Narrator
welcome to Talking technology with ATLIS, the show that plugs you into the important topics and trends for technology leaders all through a unique Independent School lens. We'll hear stories from technology directors and other special guests from the Independent School community and provide you with focus learning and deep dive topics. And now please welcome your host, Christina Lewellen.

Christina Lewellen
Hello, everyone and welcome back. We are talking tech with ATLIS. I'm Christina Llewellyn. I'm the Executive Director of the Association of technology leaders in independent schools. And I'm so glad to be joined by my co host gentlemen, how's your day going? Please introduce yourselves?

Hiram Cuevas
Yes. Good morning, Christina. This is Howard Cuevas from St. Christopher's school in Richmond, Virginia. I'm the Director of Information Systems and academic technology. Been here for 32 years and I am thrilled that we are going to have a conversation with Eric today. Me too.

Bill Stites
And I am Bill states. I am the Director of Technology at Montclair Kimberley Academy of note I am now as I was telling our other co hosts I am officially an empty nester to college students now and living the dream.

Christina Lewellen
So who's babysitting you bill? I mean, like who's making sure you go to bed on time?

Bill Stites
All I know is the going joke is everyone is fair for all for the safety of my wife.

Christina Lewellen
Do we need to invest in her mental health? Is that what we're talking about? It's important. Yes, I

Bill Stites
believe so all I know is I got home from the final drop off. I stood on my front stoop. And like Braveheart, I put my hands above my head and I yelled freedom. And it was it just was so cathartic, it felt great.

Christina Lewellen
Well, I'm so glad you've joined us on your first day of freedom, because we have a really important podcast to record. So we're getting into it. Now. We're so glad to welcome our guests today, Eric Holman. And, man, Eric, you've been making quite a name for yourself in this space, I'm sure for a very long time, but you're on everybody's lips right now. I'm gonna let you introduce yourself. But first, I just want to say that Eric, and I, and my team and ATLIS, you know, we've gotten to know each other pretty well in the last few years, because we thought that it would be a real clever idea to co host the ATLIS annual conference with this emerging bubbling group of institutional researchers. And Eric, you're the head of that crew. And so we've found that there's such amazing crossover to have the technology folks from independent schools in the same room as the institutional researchers, whatever those titles are at the schools. I know that that continues to evolve. But you know, it just makes sense that you would be in our first season of talking tech with ATLIS. So welcome to the show. And please take a moment and tell everybody who the heck you are, if they don't know already.

Eric Heilman
Oh, well, thank you. Thank you. Thank you for having me. I'm Eric Heilman. I'm the Executive Director of the Center for Institutional Research and independent schools, which is based out of Morais school in Washington, DC, although confusingly, I actually live in Denver, if my name is Andres Lipsitz, in large part due to the you find people that are in this call, ATLIS was one of the first associations that gotten contact with me when we started serious, four or five years ago. Now, I agree, it's been an amazing partnership. And it's a natural pairing, you know, Data is the lifeblood of institutional research. And so there's a lot of overlap between what the folks in my space are doing and what the folks in your space are doing. So, yeah, it's great to be here. And I'm excited to have a chat with you all.

Christina Lewellen
So I will kick things off, at least with a little bit of the kind of series of questions. I know we have a lot of them and our listeners probably have a lot of them. But can you at least just tell us, maybe a little bit about your origin story, just kind of how you fell into this work. But also what about Cirrus? Like, where did that come from? And why now?

Eric Heilman
Great question. Okay. So buckle up, kids. Picture it. It's 1995, Jacksonville, Florida.

Christina Lewellen
This is amazing. Can you please give me a soundtrack to this origin story?

Eric Heilman
It's probably something like salt and pepper back in the day. That's what was playing?

Christina Lewellen
Yes. Okay. You may continue. Yeah. Okay, thank

Eric Heilman
you. I'm in my AP Bio class. My teacher's name was Mr. Butler. Mr. Butler, if you're out there, I apologize for this story. So this is story, the only time I've ever been thrown out of class in high school. The AP Bio curriculum back then, and probably still does has a unit on statistics, you know, because people who do scientific research need to know statistics. And we've been going through this unit for almost a week, and I was in the back row of class and I was just constantly muttering petty things to my friends to be like, Z scores. What is he talking about? We would know what is this? What is this guy even talking about? No one would ever use this makes no sense. And he eventually got tired of hearing my under and sometimes over the breath commentary and booted me out of class. And I was just outraged. How dare you sir. And I was like, I have no regrets. Statistics is dumb. Who would ever need it? Turns out I'm sorry, Mr. Brother, you're right. Z scores are great. I ended up you know, I was taking calculus also at the time, but I ended up choosing my college in some maybe not small part because there wasn't No math requirement. Wow, I was in the school Foreign Service at Georgetown,

Christina Lewellen
this is revealing. This is incredible. I cannot believe this.

Eric Heilman
When I left for college, I was like, I am never taking math again. And I was studying international relations. And there was a core requirement to take economics which I had never taken before. And I was really into econ. Fast forward to junior year, I had a bunch of credits from Afra mentioned high school curriculum. And so I was able to take a leave of absence from college, and I was actually interning full time at the National Economic Council at the White House. It was the fall of 1998, which was an intriguing time to be working in the White House in general, the Star Report was dropped off at Congress. During that time, Bill Clinton's grand jury testimony was aired on CNN, The day he got impeached was my last day of my internship. So it was a wild time to be in the White House, just in general,

Christina Lewellen
we might need a whole separate podcast on that. But

Eric Heilman
yeah, that's a whole other story. But more to the point of this origin story, it was the first time was really working with practicing economists. And I really saw up close and personally, how they were using math to sort of wrap their minds around these huge policy questions, you know, education, policy, social security reform. And I was super intrigued by the work they did. But I also understood that when I tried to read their papers, I couldn't understand the modeling because it was all in math. At this point, I had not taken math for two years. And I kind of decided at that time, I was like, oh, you know, if I do ever end up wanting to pursue going into the Foreign Service or something like that, I can teach myself a lot of that stuff later in life, I'm never gonna be able to teach myself multivariable calculus or linear algebra or these other things. So I switched my major, and then went back to classes second semester, junior year, and only took math and economics for the rest of the time. And then I ended up going to grad school for economics at U Chicago, which, if people aren't aware, it's just kind of full of people who've won Nobel Prizes for their work in econ. I was a research assistant for James Heckman, who won the Nobel Prize years working on for a lot of his work around statistical techniques to sort of understand the determinants of educational attainment. Why do some people drop out of high school? Why do some people go all the way through grad school? Is it something about early childhood development? Or is it financial constraints later in life? There's a lot of statistical issues around measuring that. And he dedicated his life to it and won the price war. That's pretty incredible. Yeah, it was amazing. We got bagels that day in the office,

Christina Lewellen
Nobel Peace Prize, you get a bagel? I mean, I think at the very least you deserve a bagel.

Eric Heilman
Yeah, exactly. He didn't come visit us on our side of the office. But you know, we did get the bagels. And that was nice. But yeah, there were a lot of really just impressive economists there that really spent their whole life's work using mathematical modeling for sort of these human interactions. So you know, people hear economics, they think, oh, Wall Street and stocks. And that's not the kind of econ I was doing. Fast forward, I ended up not going into academic economics. But I'd always been interested in teaching. And so eventually, I ended up teaching math in independent schools in the DC area. I've been at marae since 2010. So I still obviously I'm working there. And so I've been teaching middle and high school math.

Christina Lewellen
And what does Mr. Butler have to say about this?

Eric Heilman
You know, hi. I don't know where he is. He's I've liked Joe and I have lost touch. As it turns out, I might not have been his favorite student. Oh, I

Christina Lewellen
said, so you got kicked out of math class. And now you're a math teacher?

Eric Heilman
Yes. Which is hilarious. Because, you know, over the years, shockingly, I had a number of students that were like, when are we ever going to use this? I hate math. I'm never going to take it again. It's like, buckle up, kids. We've got a story about that. That's awesome. I hate to break it to you. But you might end up teaching math someday, they're like, you will take that back. Mr. Heilmann?

Christina Lewellen
That's right. See, it can happen. I know. Right? But it's really interesting how like you had this big picture perspective, having come from an economics background, Nobel Prize winning leadership, working for the government, and then you end up at an independent school. And you come into this little ecosystem, this small community, was your brain always sort of knocking around the idea of layering your experience in economics and math and putting that on the human interactions of independent schools?

Eric Heilman
So yes, and here's the story about that. I would say that initially, you know, I was teaching AP statistics from early on. And you know, my first few years of teaching, and it's very much just kind of learning the AP curriculum and going through everything that was sort of prescribing there step by step, then, you know, fast forward eight or nine years, I was really able to weave back in a lot of the papers and research that economists have done, that illustrated the concepts that were in the course and actually had a really cool statistics course around sort of these statistics of the American dream. So when we talk about you know, you can just Pull yourself up by the bootstraps. And all it takes is hard work and grit, you're really examining those sorts of systems, looking at issues like intergenerational wealth accumulation, and you know, the correlation between your income and your parents income and that sort of thing. But really framing these things in a statistical way that also covered the concepts that were in the course. And so I think in my curricular design, I was definitely weaving in a lot of my experience from both the government and in grad school. And then through that course, we also started doing action research projects with the students around, you know, a typical question might come up would be, the school supports athletics for boys more than it does athletics for girls. And so we would say, Okay, well, how can we actually explore that using statistics? What kind of data would we need to collect? How would we go about answering a question? How do you form a question that can be answered? And so we started doing those kinds of projects. And also at the time, I had moved into a 10th grade Dean role. And so administratively, you know, we'd be pondering different policy questions or whatever coming up in our meetings. And just because of my background, my first instinct was always to be like, Look, do we have any data to help us

Christina Lewellen
figure this out? So you were that guy in the meeting?

Eric Heilman
I was I was, yeah, that's what you get dive there a couple math people on that team of great Dean's. So you know, you light a fire, you're gonna get warm? I don't know. And so I had done some preliminary work, probably, almost 10 years ago. Now. You know, Moray was really starting to grapple with questions of equity inclusion, there were a lot of anecdotes and narratives floating around about how different groups were performing in the school. And that's one thing. So I said, Well, why don't we actually just compile the grades into a data set and do some kind of analysis? So we can see, are there gaps? Where are they? Are they different in different departments? You know, is it changing over time? Does it depend on what division the students joined the school in. And so I sort of did a statistical study of great outcomes looking at, you know, race and gender and those sort of questions. And that was sort of the first time I did an institutional project outside of the classroom. And that planted the seed that for a lot of fruit later on, I guess, it was interesting.

Christina Lewellen
Normally, I would just kind of randomly invite bill and Hiram to jump in. But I think that what is appropriate for this podcast is Whichever one of you was kicked out of a high school math class gets to go next. Or any high school class. So let's see who steps in. No,

Eric Heilman
I cannot be the only person in this call that was ever kicked out of class.

Christina Lewellen
I mean, you might be Eric, we're learning new things all the time, or the black sheep of this podcast, you might be

Bill Stites
about the only thing I can say is I literally picked a school for the very same reason. I didn't want to touch math, I was done with it. But I was an art major. So it was a little different. So it was you know, I didn't think I was really gonna go in that direction. But lo and behold,

Christina Lewellen
Bill, you have some experience with Cirrus. Tell me a little bit about your experience working with Eric and what questions you have for him.

Bill Stites
It's very interesting, because yeah, I definitely picked up on Cirrus really at the ATLIS conference and going to some of the sessions there. You know, we've talked about this on previous episodes. But Hiram and I have done, you know, good deal of work of working with other schools, helping them go in and do like data systems audits and see if they're ready to kind of gather and store the types of data that they're going to need to kind of address and answer some of those questions that Eric is bringing up, kind of picked up on that and then became this serious junkie, when I was signing up for you know, over the course of this past year, Eric kind of organized with Sarah implying these monthly meetups where we were talking about data culture, which was great. And that was really one of the questions I wanted to ask you, Eric, it was around, like, how did you develop that culture at marae? To start having those conversations? And I'll get to that one second. But the other one was this around what I'm gonna watch Eric's face. So you all can see this cringe when I refer to Google look or studio, because you know, it's the creepiest name you could possibly give to a tool. It was this balance of using the cultural aspects and then using the technical aspects. Because for us here, MK, we had someone who wasn't officially in the role of institutional research, but really did the work. And that person left. And I was kind of like, I need to invest myself in this so that we don't lose steam, because it was, you know, to a large degree, and Eric can talk more about this, again, just kind of having spent time with him knowing that he's got some of these answers. But having that group that was working on it, or that individual working on it, you know, where were you then end up going? And then that proceeded into joining Eric this summer for doing some work on a project, which again, I'm sure he will share more about. But Eric, going back that cultural aspect, how did that develop for you at Moray, where you had this extensive background that you were talking about around, you know, economics in this work that you've done, how did that cultural aspect develop at the school? Because that can also often be the hardest hurdle to kind of overcome?

Eric Heilman
Yeah, that's such a great question. And I want to emphasize that a lot of people when they come and ask me about institutional research, and how do we get a program up and running, the first things are asked me about our who should be in this role, what kind of statistics they need to know, should we use Tableau versus Google Looker studio versus, you know, Power BI? And those are all good questions. But I always encourage them to think and we talked about this in the guide, is your culture ready for institutional research? First, because you know, the long term success of an institutional research program is, I want to say maybe even more determined by the cultural setting of the school than just the skill set and platform knowledge of the person who's actually doing the work. We had a whole professional learning group last year on data culture. And I think, you know, we talk about data culture, as being an openness to change, and relying on trust between people in a community, I think that the narratives people have about themselves and about the school community that they're in, can be very personal and be very core to people's identity. And so when you're introducing a new feedback loop, which a lot of IR data feedback systems are, that can potentially produce counter narratives, or shift the way we understand things that can make people uncomfortable, particularly I think, people who've been in a school for a very long time, you know, they feel like they have it all sort of figured out, and they understand what's going on all around them. And so when you present evidence, or there's an indication that things might be different, or maybe the reality has shifted from what it was in the past, and so people's beliefs aren't up to date, that can feel disorienting to people. And so I think, making sure that you're in a school community where people feel that feedback is actually valuable, and constructive and positive. And not necessarily attacking or belittling is incredibly important. So, you know, in the guide, we talked about the precursors for an institutional research program. And in there, we talked about things like, what's the role of professional development in the school, you know, is there an expectation that people are already going to be learning about new things, what's going on how to improve their practice in whatever area they're in, in the school, and also a feeling of trust that people can get feedback from their supervisors also give feedback about their supervisors in a safe way, and that that relationship is constructive?

Christina Lewellen
Hey, Eric, was to some magical guide that you keep talking about? Why thank you for asking, cuz it sounds like people need to get their hands on this.

Eric Heilman
Yeah. So every summer serious hosts, the serious summer fellows lab, and we bring together a group of thought leaders around a specific topic area, and we pay a stipend and their travel expenses to come to DC to help sort of accelerate their institutional research projects. In the previous years, we've done topics like holistic measures of student engagement, we've done the measures of equity inclusion in school communities, we've done a summer lab on Admissions and Enrollment Management, institutional research. And then for our fourth lab, this past summer, we got together a group of all stars. So some people who've been in previous cohorts of the lab and some new folks to really generate and write a guidebook for schools that are looking to start an institutional research program, we looked at the questions that we get most often from schools, and put together sort of the answers to all those questions that we tend to get most frequently. So the guide is coming out Well, right now it's late August. So within a week or two, it should be out. That's exciting. Yeah, I know, I'm very excited.

Christina Lewellen
We'll put a link in the show notes. So once this drops, everybody will be able to go find this magical guide. Awesome. And you'll be hopefully be long past the finish line.

Eric Heilman
Every finish line is just a start line for something else, you know, truth? Yeah, the guide is coming out. It addresses some big picture questions that are useful for both actual IR practitioners and for school leaders and board chairs and that sort of thing. That's where we're looking at things like how do schools structure the role who's doing this work, issues around data culture, and then the second half of the guide is really there for folks who are out there doing IR work. So there are chapters about what statistics knowledge do you need to have in order to do this job? There are chapters on data visualization, what's the best way to present a report or dashboard, graphic design elements such a thing? There's a chapter on designing surveys. So all the technical and practical pieces are in there as well. Bill worked on it, so it must be amazing. I think we're all really hopeful that people will find it useful and it will be be a resource for folks.

Christina Lewellen
Yeah, I mean, it's such an emerging piece of the Independent School puzzle, that I think that the fact that you've come together to share what you've learned so far, it's a very evolving, organically growing thing. And so to have something people can kind of stick a pin in is probably such an incredible resource. I'm really happy for you that you guys have gotten that out. Oh, thank you. Thank you.

Hiram Cuevas
So Eric, I'm already thinking ahead to summer of 24. Obviously, we'll wait until the guide is complete. But I'm sure people's interests have already been piqued by the fact that you have these cohorts and may may express some interest. Do you have the theme already outlined for summer 24?

Eric Heilman
No, not yet. Usually, we decide on a theme in early January. And so most of August through December through our different serious webinars and meetings and gatherings. I'm just kind of keeping my ear to the ground and seeing where people's interests are. What are the issues that are cropping up this academic year? Where does the need seem to be in the community and then we pick a topic based on that in early January. And then you know, applications are open, I think through the end of February. And then the cohort actually starts doing pre work with me and with each other in the spring leading up into the summer. But during the year Cirrus offers a number of professional learning groups and collaborations with associations to provide PD for people. So last year, we ran two year long professional learning groups, one on data culture, and one to teach people to use begin the year was called Google Data Studio. And then by the end of the year, they had rebranded the Google Looker studio, which another person in the cohort Camila Calkins, it Lakeside Kristin did Google creeper studio. It's like, yeah, that sounds about right.

Christina Lewellen
I mean, I kind of wish I was in the room when that name was being kicked around.

Eric Heilman
I actually think it might have been at the conference, Christina. So you were at least in the building?

Christina Lewellen
I mean, like if they came out with looker. Like if that was the best name to emerge, I kind of wonder what else was on that whiteboard? You know what I'm saying? Cheese?

Eric Heilman
I don't know, who focus group this. Exactly. I did actually hear that Google acquired a company called looker. And that's why it's in the title that might be an urban myth. I don't know.

Christina Lewellen
You know what, we're not about facts on this podcast. We're just gonna spread the information as we know it right.

Eric Heilman
In the modern era, doesn't matter if it's true. It matters. It's a good story. So yeah.

Christina Lewellen
It's kind of true. So Eric, let me ask you about some of the AI that might touch on the work that you're doing with data with research. Does it affect how you guys work? How you think? I mean, I think that it kind of touches everybody. And certainly independent schools. We're talking about AI from the perspective of the classroom and also some productivity hacks, being very aware of the privacy issues. But are you in the institutional research community? Talking about AI? What do you think about it any potential their thoughts? Oh, yeah.

Eric Heilman
So in the guide, we have a blurb. I think the blurb isn't titled something. Speculation about the future of AI in institutional research. So yeah, there has been a ton of conversation about sort of the classroom and student experience side of AI. I think, for data analysis, there's a lot of exciting possibilities that might be coming down the line here, I'd say the first way in which I see AI, simplifying the work of people doing institutional research work, is in qualitative coding of open ended survey questions. So a lot of times, you have schools that are doing surveys, they want to have a question where people can just type in whatever they want, how was your year, this year or whatever? Doing good analysis on text answers is extremely time consuming and takes a fair amount of skill. And with the advent of these natural language models, you know, we're getting to a place where you can have an AI read those comments and automatically find the themes and code them. And it cuts the time it takes to do that kind of analysis by 100. If you can figure out how to connect yo chatty Vt to those and also mask all the PII.

Christina Lewellen
Right, that would have been my question, how do you protect your data, then? Yeah,

Eric Heilman
yes, exactly. There are some folks in our space that are writing scripts in Python to identify and mask all the PII, then they feed that into chat GPT to do the qualitative coding. And so it's a really intriguing area. Most schools don't have the staffing to do a lot of good qualitative coding. And so they might steer away from asking those questions, which is shame, because they are super useful if you have the staff power for it. That's the first way I think AI will come in. And then I think there's this longer term horizon with AI that has to do with our data systems. I think there is a future in which, you know, schools are recording data more and more frequently about everybody on campus, whether it's online, great books, right. It used to be that teachers would push out grades to parents and students, maybe four times a year, you know, to progress reports to report cards, but now with the micro level data that we're accusing Waiting in these live grade books, you know, where people are putting in individual assignment grades that are updating daily, where people are checking in and out of the health office. And that's being recorded digitally, you know, there are all these little pieces of data around campus, that with the right pattern recognition, you know, could help us understand and recognize when people need support earlier, whether it's students or employees. But that's going to require a different orientation towards how we keep data. I think one of the biggest obstacles that Indian schools face and really getting into AI, it's just, our datasets tend to be fragmented. They're in different databases, they're inconsistent, as role shift people change the way they notate data that they're inputting. Data Governance is not always super clear. So therefore, the data may not be very clean, which is fine in most schools, because most schools are really focused on using the data to complete a task, you know, we are recording this data in order to decide who to admit or decline into our school, we are collecting this data to produce a report card or a transcript. And all those tasks are happening. There's this other level of insight oriented data that I think schools are just now getting to where we're not just collecting data to complete X tasks. We're using the data we've accumulated through all this to answer larger questions about our community experience.

Bill Stites
Hearing you say all that, and starting with the AI piece, and I was thinking about, like, when we were around the table this past summer, just thinking about like some of the stuff Hudson was showing us at the end, one of the members of the group this summer, Hudson Harper, he was showing us this intricate way of coding this with AI on the back end, and it was like, your mind explodes, you know, you think of a gift of the guy's head. But that's exactly what it was like. But as you talk through what you're describing here, this is a role that, you know, has a very wide breadth of skills that are needed. And I think back to the sand, you know, like it takes a village, what do you see as some of the ways in which either an individual or individuals or a school could kind of structure this within their institution to really build on all of the key pieces that you need to have in place to really have a thriving IR program?

Eric Heilman
Yeah. So we covered this in the chapter that Bill and Hudson did such a nice job of waiting. I know. Yes. So there's a chapter on data strategy. And so we talked about data strategy as a combination of data infrastructure, data governance, and then the strategic piece to it.

Christina Lewellen
Eric, can I interrupt for just a second and ask what's data governance? Like for those who are just really not in this space? Can you just quickly define that, as you answer Bill's question?

Eric Heilman
Yeah. So in the guide, we use this analogy, data infrastructure is like the roads and highways that our vehicles travel on data are the vehicles, right. So data governance, would be similar to the rules about how the cars need to be built and constructed and the standards, they need to be up to the rules about how fast things move from here to there within the data infrastructure. So data governance really is a set of policies that look at the quality of the data you're putting in. So for example, if you're an admissions office, there should be a data governance policy about how you name feeder schools, when they go into your system. So St. Joe's, college prep, middle school, or whatever, should always be called the same thing. But in a lot of schools, we see sometimes it's called St. Joe's, sometimes it's St. Jays, sometimes it's Joe's Middle School.

Christina Lewellen
That's a great explanation. That's a very simple way to kind of boil it down. Yeah. And

Eric Heilman
then data strategy is really stepping back and thinking, are these roads? And are these vehicles actually going where we want them to go? Are they serving us? Well, more of like an urban planning level of things is our system doing what we want it to do and measuring the things that matter to us. And so those three elements together, we recommend, if you have a single person that is somehow magically an expert in all of these areas, from database architecture, to data governance, to philosophically aligning your metrics to the mission and your strategic plan to the actual statistical analysis to building that? That's a lot, right? It's very rare to find a single person do that. So we recommend that schools put together a standing data strategy committee that's made up of the IR person and director of tech, and then the primary data custodians from around campus. So the registrar, admissions development, and somebody from the senior leadership team. And the committee is charged with taking care of all these elements. So, you know, they work together to make sure that the data governance policy works for the data they're collecting, that they are collecting the data they need to be collecting in order to answer the questions that are important. So if one of the school's strategic priorities is wellness, or equity inclusion, are you collecting the data that you need in order to actually measure those things? And if not, how are you going to design a system that can collect it and how is it in a way that you can integrate into the rest of Have your data ecosystem. And so those are the kinds of things that, you know, a data strategy committee can shepherd through school and really take advantage of collective knowledge. And so depending on one person to know it all,

Christina Lewellen
I mean, that goes to Bill's question, right? Like, it's complicated. Are there schools doing this with one person? Like one data cop?

Eric Heilman
Yeah, I think I might have seen a code. But I would say it is a little bit easier at a smaller school, just because there's less going on. Yeah. And so it might be more accomplishable. In some smaller schools by it. I've seen some places where there might be one person, but they're working in a specific area of the school. So they are doing IR just for the admissions office. So they're really kind of managing that sphere. And that's doable for one person, but to really have a full breadth of an IR program that does, I think, take a village for sure.

Hiram Cuevas
I think what's interesting, the database audits that Bill and I have been on one of the challenges I think schools have are these siloed systems. And oftentimes, people have to give a little in order to for the entire school to actually benefit by the data governance that you're trying to deploy. Most departments when we're dealing with these schools that have the siloed environments, they don't want to give up anything, because they think theirs is the only way to do it. Yes. And consequently, the school can't benefit from everybody talking the same language.

Eric Heilman
Correct? Yeah, in the guide, we talked about, you know, one of the most important things to understand institutionally is that the data that exists in these silos is an institutional asset. There is no office or person that owns that data, that data belongs to the school. And I think one of the benefits of having people work together on a data strategy committee is that it helps people in the Registrar's Office understand why the data coming in from the admissions office is the way it is, and they can hear each other talk about well, I need it to be in a slightly different form, because we want to use that data later on. To try to connect, for example, data we collect from applications to actual outcomes once kids are, you know, actually come to the school. So that you can actually study like, is the information that we are collecting through applications and making decisions on actually correlated with outcomes later on. But in order to do a project like that, there needs to be some cooperation in terms of the data and the data quality and what you're collecting between those two offices. And I think once people hear why we might need to make changes to governance, they get on board, because if there's one thing we know about people that work in images, we really care deeply about the students and our colleagues. And so if there's a compelling reason to be to change the way we're typing and data or validating data, as long as we understand that it's for the betterment of our community, and the people that we care about, I think people can get behind it.

Christina Lewellen
So I know you mentioned Dei, but can I ask all three of you, what is bubbling up as the data that either your school or most schools are trying to track right now what's hot in the world of we should track this, because

Bill Stites
so one of the areas that we've been focused on a lot here, outside of just the pure academic data that we have, is exactly what you've just brought up, because it's around some of the DEI work that we're doing at the school. And what I'll say about that is it'll go back to what we were just talking about in terms of how are we using the data? How are we reporting on the data? What are we telling our constituencies, you know, our parents, our students, our faculty, our alums, whoever we're gathering this data from, why we're collecting it, what we plan on doing with it, and then once we say we're going to do a thing, how we are then going to share that back out. So at the last ATLIS conference, I gave a session on just how we at m k kind of structured this, because you need to start looking at how schools need to start looking at all of their systems. Because, you know, Eric mentioned admissions, you know, you look at an admissions form. And if you say that you're going to give incoming students the ability to pick their own pronouns, or be able to choose a gender, that is something other than the binary, male, female, if you go with a non binary option, or some other options that you might want to express there. But then you get into your interest and you start listing boys and girls soccer, you've kind of just kind of had everything that you just said that you were respecting based on the ways in which your forms are looking at. So thinking about the ways in which you're structuring the data, and then how do you use that information once that gets in? But then how does that information then go from? In our case, we're using bare cross to gather all this information. Well, how does this end up in our development system? If we're using a tool like you know, razor's edge, if all of those fields don't go there? So there's this conversation about how do you structure the data? Where is that data of authoritative? All of those things just so that we can begin to address and answer questions that are stemming from those data points. That's where I see a lot A lot of the conversations we're having right now are focused on.

Hiram Cuevas
And I would say we've just completed our strategic plan. And I feel like I've been stalking Eric for quite some time now.

Eric Heilman
Was that you outside my window last night higher? What?

Hiram Cuevas
Well, it's been interesting for us is we've looked at Moraes students thrive bottle. And if you haven't had an opportunity to take a look at that thrive model, it's really something that interested us, because we're looking at the portrait of a graduate, which is a very similar concept among our among schools, like ours. But we wanted to target some of the things that we were also looking at from a strategic planning perspective, the things that we had covered. And so seeing the work that Eric has done with that student Thrive model, we were just like, oh, it gave us a whole new perspective on how to actually attack this particular problem. And now we're engaging in the conversations to make it much more meaningful based on the information that we have on site. And I'll post a link to that in the chat notes.

Eric Heilman
Yeah, I was gonna say the exact same thing. So without going into the long history background, the Thrive model marae was something that I sort of designed and set up in response to dei questions, right. As I mentioned earlier, I had done some analysis of great outcomes already. And so we were aware of places where there were discrepancies and average grades. And then we started doing annual alumni surveys. So we survey our alums two years out, and five years out after graduation. And what we started finding is that some of the groups that had the lowest average grades, while they were at the school, once they were graduated, often had the most positive feedback about their experience at the school overall, once they were out. And so to me, that almost felt like a paradox. And anytime I'm doing analysis work, and I encounter a paradox, I get very excited, because that's where interesting stuff happens. And so that's when I started thinking, Oh, well, clearly schools about a lot more than just grades that's in our mission statement. And every student knows that every teacher knows that. But how are we actually getting at measuring the experience and all these other areas besides just great? Yeah, that's when we sort of put together Thrive model at Moray, we have six dimensions that we measure there straight out of our mission. So we've got two academic dimensions, one of which is looking just at grades. The second is looking at how students feel about their learning and their progress as a student, then we have dimensions for athletics, arts, social community connection, and then physical and emotional health. And so we compile all the data that we have, from the school side about each student. And then also, the students themselves fill out a reflection survey every semester about their experience that semester. And we combine all that data together to create a metric in each dimension for every student, which we can then aggregate up into average profiles. So we can aggregate the individual student profiles, by gender, by race, by grade level by division of entry. And that helps us answer questions like, Is there an experience gap in social community connection across race, not just grades, so we can detect gaps. And then we actually have a framework through which we can gauge progress in improving equity inclusion in the community. And I think a lot of schools get stumped on that, you know, they sense that something might be off, or sort of anecdotally, they know from the community that there are issues in certain areas, and they try some interventions. But later on, when they're like, did it work? You know, they're kind of left with gauging progress by like, I think it feels like it's battle, you know. And so, for us, it's a way that we can ground some of our discussions and try to think about whether or not we think we're headed in the right direction.

Bill Stites
Yeah, I think that's all because it gives you the ability to not just report on data, but use data in almost like a predictive sense. You know, we talked about how AI might help with some of these things. This is taking something that is independent of even an AI model, and applying that so that you can use your data to inform decisions, and really focus in on those students of need in those areas of need. One of the things we talked about during the summer, but one of the things, you know, that I think can often come up and we were talking about, oh, well, you want to do this or this or this. When you think about the questions that come up, and all the different questions that come up, who gets to say yes or no to the questions because, you know, somebody or committee or an individual that is in this IR role, I could come and say, I'd love to know this, or I'd love to know that. But is that really aligning with mission and goal and those things that you're talking about? So how do you see schools, you know, addressing the question of questions.

Eric Heilman
That's a great question about the question of questions. Let's see how far we could take this. And now I sound like a broken record. But this is something we talked about in the guide.

Christina Lewellen
Magic guide. Yeah.

Eric Heilman
So it works differently in different schools right now. So we did an institutional research state of the industry survey as part of the preparation for the guide and the results of that survey are also included. In the guide, so one of the questions we asked on the survey was how are projects selected? So basically, who gets to ask the question, and the vast majority of respondents said that the head of school determines what the project should be. What we don't know is how heads of school are prioritizing projects, I assume they're not just throwing a dart at a dartboard and choosing there, there's probably some sort of discussion amongst the senior leadership team, that sort of thing. And so that would be interesting to dig into. But it does, for the most part seem to be, you know, the head of school is making a decision and choosing projects that are most relevant and strategically important for the school at the time. And usually, that's, you know, something connected to a strategic plan coming off in something like that, or accreditation, self study, and that sort of thing. There are definitely other places. So San Francisco University, city, high school has this really cool model, where they have a sort of standing committee for IR. So there are a couple permanent members of their committee that include an assistant head of school. Nassif, if you're out there, hello. And then a couple of members of like the math and science department that have statistics knowledge, but then faculty members can apply and propose projects. And if their project is selected, they rotate on to the committee for a semester. And they actually get course relief in order to participate in that. And so there are models out there where the project selection, and the question selection is a little bit more open and coming from the ground up. So in the guide, we suggest that both of these kinds of projects are really valuable, particularly to help foster a constructive data culture. So data culture often develops best if the whole community feels like they have some ownership in the IR program. And so having at least some venue for projects that are coming from the ground up rather than the top down, I think can be really valuable. So we suggest a blend, like definitely, we want schools to be strategic, and invest in IR projects that are very related to the mission. But sometimes those projects can feel kind of removed from the day to day life of what we are doing in schools. And so having some flow of projects that are really focused on sort of the nuts and bolts of what we're doing in the classroom, and in our offices, I think is great as well.

Christina Lewellen
It sounds like there's so much like we've mentioned culture a couple times and buy in is basically kind of what you're talking about that like data can't you guys can't just go hide in a closet and come out with findings, right? Like there has to be a little bit of an effort to make sure people know what you're doing.

Eric Heilman
Oh, my God. I mean, you could, but you wouldn't last long. So it's interesting, you know, the story I told earlier about that first project he did about great analysis. You know, I learned this through experience, when I was discussing some of the results from that project and a great Dean's meeting. The reaction was, it was emphatic. There were a number of meetings that happened almost immediately in the head of schools office around like, who has seen this? You know, that's a culture issue, right. And even though I thought I had explained what the project was about, and what the outcomes would be, I'm just not sure we had talked that through or that I had thought through or anybody had really thought through. How is this going to feel to people when it comes out to me because of my background as like, this is just like a normal data analysis project. And I've only been at the school for two years. So I was in a very different place.

Christina Lewellen
So who's this new guy with the math? Right, right.

Eric Heilman
Exactly. Exactly. And is he flammable? Can

Christina Lewellen
we can we eliminate this situation? Yeah, exactly. Exactly.

Eric Heilman
Is there a steak around here? I would say, Murray has come a long way since then. But that's really the first time that I started thinking about how important that cultural piece is.

Christina Lewellen
It's important. Yeah. And for those who, like haven't walked the path. Okay, so you guys have been on your journey, you're probably ahead of some schools, if a listener is enjoying this conversation, and also thinking about getting involved in institutional research in some way, shape, or form. Where do they start? Yeah, it's overwhelming. Can you guess where we discuss it? In the guide? It isn't the guy. Yeah. So go find Cirrus and go find the guide.

Eric Heilman
Yeah, so that's the short answer, I'd say a slightly longer answer. First, I think every school is already doing some form of IR on our website, and we say x percentage of the faculty have an advanced degree or y percentage student body or students of color that is descriptive institutional research work. We're putting data into Dazzle, you know, we produce board reports, we are doing some quantitative descriptions of our community already. So every school has already started on the journey. We always encourage schools that are just starting out to focus on what we call low hanging fruit. And so for us, we recommend two types of projects to start off with, one we call process simplification projects. Those aren't necessarily where you're doing some sort of deep analysis or that's where the that's where somebody who's good with data and maybe dashboarding listens to people. And here's what are your pain points throughout the year. Maybe it's making the schedule every spring, maybe its parent teacher conference, logistics, so finding some sort of logistical and finding a way to make that easier for people and that helps the institutional researcher practice their data skills and sort of their back end structuring, but also wins a lot of goodwill in the community, particularly if you are helping do these simplification projects with some of your data custodians around campus that may need to change their governance practices a little bit later on. And I'd say the other type of project that schools sort of dive into first are what we call descriptive IR projects. So this is where you might be visualizing your grades, grades and admission are great places to start, because that's typically where we have our most consistent and cleanest datasets. So, you know, looking at an analysis of great outcomes is usually a pretty accessible place to start, or, you know, looking at admissions numbers, funnel metrics, yield determinants, that sort of thing in admissions, enrollment management, well, we discourage schools from and this goes back to when the question was asked earlier, a lot of schools want to jump immediately to these things like the Thrive model, or things that are measuring wellness or equity, inclusion, or engagement. Those are really, really tricky and require a lot of skill on the back end, both in terms of data infrastructure, as well as mathematical modeling, that's in the statistics behind it. And then also the cultural piece, you know, if people are having feelings about comparing average grades, that are already numbers across groups, you can imagine culturally how it feels to be talking about gaps or discrepancies that might exist in emotional health. We always say, Okay, try some of these more accessible and you know, projects that will win some goodwill before she started expensing on these other types of projects, and

Bill Stites
listening to all this in terms of the way in which you're talking about, like the projects that take on and all those other pieces and how you get this work going. And just want to put in a quick plug. We just recorded a little while ago conversation with Jim bologna about how to set up committees, how to build the structures and the buy in for getting those things locked. So if anyone can generally do this conversation, once they really dig in on what that process looks like, I just encourage them to give that episode a listen, because I think there's some great pieces there that apply to how you can get a lot of this work going and getting those committees organized and getting the buy in that you need. So I just wanted to throw that out there for everyone.

Christina Lewellen
Yeah, great plug. Well, as we start wrapping up this episode, which has been kind of mind blowing in a lot of ways, learned a lot about Eric learned a lot about Cirrus and data. And we anticipate that we'll have another really powerful lineup at the ATLIS Annual Conference, which will be happening in the first week of April in Reno this year. It's been such a real pleasure working with Cirrus because opening up that institutional research track and allowing you to fill that time with the valuable conversations that schools need to be having around IR. It's really been so rewarding. And I think our members love it. And I think our attendees love it. And it's just been a really lovely partnership. So I very much appreciate that. And we'll look forward to another great year of content with that.

Eric Heilman
Christina, we can't forget to mention the happy hours as well.

Christina Lewellen
There's some good happy hours going on. Apparently the data people know how to party. You know, as much Eric as I adore you and I really look up to you and I value all the work you're doing. I do think that it's really important to make a note that I actually follow your dog on Instagram. So Boomer is this gorgeous beast? And so can you just tell everyone why I'm obsessed with Boomer?

Eric Heilman
I assume it's because he's beautiful.

Christina Lewellen
He's so beautiful. He's huge.

Eric Heilman
And nobody knows how to take a nap like him. He is truly a role model.

Christina Lewellen
So if others are interested in following Boomer on Insta, can we drop that in the show notes? Like is it okay, if he goes viral?

Eric Heilman
I think that'd be great. He's not great with spreadsheets, I have to tell you, but he will support you while you work on your spreadsheet.

Christina Lewellen
Wonderful. Beautiful bernese mountain dog. He is fun to follow. My kids love him too. Like the whole family follows. Well, Eric, thank you so much for joining us on the podcast. We're really glad to have had this conversation. You are welcome back anytime. Oh, thank you. And maybe we can spend some future episodes diving into some of these pieces because it is a lot. But in the meantime, we'll make sure that everyone in the show notes knows how to connect with you how to connect in with Sirius and how to get their hands on the magical guide. So thank you so much for being a part of this

Bill Stites
guy. So left Avenue. Thank you very much.

Eric Heilman
Thanks. It's great to hang out with you all anytime.

Narrator
This has been talking technology with ATLIS produced by the Association of technology leaders in independent schools. For more information about ATLIS and ATLIS membership, please visit the ATLIS.org If you enjoyed this discussion, please subscribe, leave a review and share this podcast with your colleagues in the independent school community. Thank you for listening.