Dave Rush LIVE Q&A With Greg Kurtzer and Skip Grube

Join Dave Rush LIVE as he talks about using Raspberry Pi to learn CompTIA certifications and computer technology. Dave welcomes Greg Kurtzer (founder of Caos, CentOS and Rocky Linux) and Skip Grube (maintains Rocky image for Raspberry Pi) to talk about FOSS distro development, upstream/downstream devs, RPi and lots more. We'll do Q&A, talk about studying technology and CompTIA exams using Raspberry Pi computers. Big news: there is a contest for access to your choice of a TotalTester Online (TTO) practice test! Amazing discounts on Total Seminars certification prep resources.

Webinar Synopsis:

• What is Red Hat?

• Building Rocky Linux

• Where CentOS Got Its Name

• Upstream Vs. Downstream Distro

• Raspberry Pi

• Rocky Enterprise Software Foundation

• RESF Support Resources

• Rocky Linux to Distribution

• Prioritizing What to Add

• Red Hat Enterprise Linux

• EPEL

• How Do We Characterize AlmaLinux?

• FOSS Support

• CIQ Educational Program

• Where Are You From?

• Working as a Team

• Chat Rocky Linux

Speakers:

• David Rush, Senior Instructor, Total Seminars

  • Linkedin

• Gregory Kurtzer, Founder of Rocky Linux, Singularity/Apptainer, Warewulf, CentOS, and CEO of CIQ

  • Linkedin

• Skip Grube, Senior Linux Engineer, CIQ

  • Linkedin

---

Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Full Webinar Transcript:

David Rush:

We are live. Happy Friday. Happy Pie day Friday. Happy drama. Welcome everybody, as you can see, as we have planned and have been announcing all over the place, we've got a couple special guests today. I'm so excited, I can't speak, there's not many people here. I'm going to do intros now, but we're probably going to do more intros again later. What do we got here? Oh, wow. We got more than the usual start, everybody's excited about this. Let me look at the actual feed. There we go. Okay. It mimics here. I hate Zoom, if I look at my Zoom page, I'm in the upper right hand corner and Greg is below me and it's completely scrambled on the output. At any rate, with the green background is Greg Kurtzer.

I won't even begin to list all of his industry credentials, but I will drag them out of them as we go on. Greg, at the moment, is the founder of Rocky Linux Group, organization, whatever. Skip Grube, almost pale in black and white tones today. From our behind the scenes prep discussion, everything that I had to introduce about you, I am now throwing out the window. Because I pretty much had you pegged as a coder/administrator. Now you've just thrown me for a curve. I'm going to open with Greg. Greg, introduce yourself. Tell us a little bit about yourself and why I dragged you here.

Greg Kurtzer:

Hi everybody, I'm Greg. I've been really fortunate in my career to have been part of a number of different open source projects and organizations. I spent a long majority of my career working at the US Department of Energy National Labs and joint appointment to UCOP and UC Berkeley. I've been just very fortunate and lucky in what I've been able to do and what I've been able to be part of. Now, I'm founder and CEO of this little company called CIQ. And an awesome Linux distribution called Rocky Linux and which I'm part of. I'm really looking forward to telling everybody and talking with you more about it.

David Rush:

Outstanding, thank you for coming. Kurt. Kurt, sorry, Greg.

Greg Kurtzer:

Oh my goodness. Flashback to high school.

David Rush:

I knew it was about to happen.

Greg Kurtzer:

All of my teachers, every one of my teachers used to get me confused with another guy in my class called Kurt. We were both just Kurt, so you just totally gave me flashbacks of the early nineties, late eighties. Oh my goodness.

David Rush:

I apologize. I've been typing your name in documents all over the place for the last three weeks and it's the Kurtzer part that comes through. Glad to bring back a memory. Welcome, Skip. The titles that I had for you basically led me to show coding. I know that your background is in administrating Linux systems. You moved on from there and became coder extraordinaire and package manager. But I didn't even know what some of those terms mean. Fill us in.

Skip Grube:

The way I got into this whole thing, we'll tell the story later. It's quite a story. But suffice to say, I was very, very early on, hopped on the Rocky Linux project, got very excited about it. Just started contributing to it more and more and more. A lot of late nights, a lot of packaging work, troubleshooting work. I don't really do much of the infrastructure, I'm out of that game at the moment anyway. But there's coding involved, there's all kinds of things involved. Eventually, just recently, actually in the last six months got picked up by CIQ. I'm now a CIQ employee, so I get to do it in part and get paid for it. It's very nice, very good for me.

David Rush:

All the contributions that you've been doing with the massive amount of projects that I've looked up on you, you pretty much did that on your own time as a voluntary thing?

Skip Grube:

Been in the open source world for a long time, not as long as Greg. Greg's definitely got me beat.

Greg Kurtzer:

It's age Skip. It's just because I'm old.

Skip Grube:

I'll become old and crotchety eventually. Definitely just, I'm already familiar obviously with the Red Hat world and the RPM world and hopped onto the project. I've learned an awful lot. It's part of why I'm working on it, quite frankly, is that I would say my day job just wasn't doing it for me. Some of the folks we work with, like technically wise, would blow you away. Like, I'd put them up against anyone in the world, quite frankly for some of this stuff.

What is Red Hat? [5:29]

David Rush:

You've opened up the door to the background that not everybody is familiar with, myself included. You said you're familiar with the Red Hat world and Rocky is certainly tied to Red Hat and your precursor CentOS and the CAOS prior to that. There are terms like upstream development and downstream development. Here we go. Give us a primer on what Red Hat and what upstream and downstream means and where Rocky fits into all that. That's an easy one.

Skip Grube:

You want to take this one, Greg? Or would you like me to hop in?

David Rush:

You know what you keep going. I like hearing you talk.

Skip Grube:

I'm going to start the story with Rocky first and then we'll talk about upstream and downstream and all that stuff. A lot of folks still not familiar with this, but I hope a lot of the audience has heard of CentOS, Linux, you're familiar with it.

David Rush:

Looking at a guy here on our chat feed, Will Shaw. One of the biggest proponents of CentOS you'll ever run across. It's walking here. I know he wants to dig into the world with you.

Skip Grube:

Absolutely. I’m also a big fan of CentOS Linux. Well, love it, loved it I guess. If you want to call it that. It all started in December of 2020, there was an announcement. I followed Tech News, I was just a member of the general public, I would say that's when there was this big tech announcement that came up that said, oh by the way, CentOS 8, which is released in 2019. Was slated to be supported with updates until 2029. Same as Red Hat, because they follow the Red Hat release, have for decades now. They said, oh, this release basically from the CentOS blog says, no, we're going to end in 2021 actually.

December 31st, 2021, after that, no more CentOS 8 updates. Yes, Will, CentOS 7 updated till 2024. Got to look it at upgrading soon. It's bizarre, CentOS 7 continues to 2024. CentOS 8, nope, nothing doing. I was like, oh, that's not great because I love CentOS and what there's and CentOS Stream, which we'll get into in a second, acts as like what Red Hat now comes from, but we'll talk about that in a minute. That's going to be the only CentOS.

I looked at this blog thing. Yeah. Okay, there you go. Yep. actually, it's still there. If you search for GMK. I was looking at the comments because it's something I do sometimes God help me. But I saw this thing that says, I am considering another rebuild of REL and may be able to hire some people for the effort and you can come join us in our Slack link in HPCNG. I was looking, looking at it and was like, huh, huh, that might be cool. I went over to the website and I know that I hopped in probably about six or seven hours after that blog comment was made.

I had never even joined a Slack before. I knew what it was, but I hopped into this Slack channel that he had posted there, and there were 5,000 people in the Slack channel and they were all talking at once. It was chaos. Like it was just everyone was like, right, what are we going to CentOS is going, what are we going to do? What are we going to do? It was almost like a crowd stream of thought. It was going a mile a minute down the screen. You should have seen it. It was first thing we did was organize different channels for different things. Because with one channel, it was like, you'd step away for like 30 seconds and you'd come back and the conversation you were following, it was three pages up.

If you heard Twitch plays Pokemon, it was Twitch Designs and operating system. It was nuts. From there it was very organic, honestly. We quickly settled that hey, the hive mind, I'm going to call it said, Hey, we got to replace CentOS, because CentOS people want this and people rely on it and we like it, so let's do it. We divvied ourselves up into different channels. People were like, well, I want to do testing. I'll create the website for it. The reason I ended up where I was actually is because I quickly caught onto this idea and was excited about it. I noticed that in the different channels there's a lot of people that wanted to do testing.

A lot of people that wanted to design the logo and to troubleshoot. But there were not as many people in the develop and package channel. I was like, well, I know a little bit about that, so I'll join that. There were a lot of people that wanted to do infrastructure like web servers and stuff for it. Because we have a lot admines so forth in the group. I was like, man, there's not that many people that actually want to do the engineering work because it's quite frankly, it sounds easy. Oh, you just take the Red Hat sources and rebuild them. But I'm here to tell you it's more complicated than it sounds. There's a lot that goes into it. That's how I set off on this wild journey. It was pretty crazy.

David Rush:

When you say package management, from what I've come to learn that sort of exists in two realms. One is the core packages, the core OS packages, the development chains and things like that. Then there's the ancillary packages like porting or changing or modifying Apache 2 so that it works on Rocky or whatever flavor, and then it can go in your repositories. Do I have that roughly right?

Skip Grube:

Kind of, sort of. I'm going to have Greg chime in here soon too. I can tell he has stuff that he wants to talk about as well. The core mission of Rocky Linux, like CentOS before us, is to faithfully recreate the binaries for Red Hat Enterprise Linux. Version for version, pound for pound as closely and most compatible as we can possibly get it. Our entire goal is that every piece of software, if you compile it for Red Hat Enterprise, Linux 8.7 or whatever, it should also work on Rocky Linux 8.7 without modification, no matter what. If there's a difference between them, we consider that a bug.

Even if the Rocky version is fixed, if it's somehow better. We don't actually want that. We actually want to be a 100% compatible all the way through. Even that is difficult quite frankly, because there's more than 3000 packages and you have to build them all in some of them in very particular ways in a very particular order. There's all kinds of messiness that gets introduced. But the other side of it is what we call special interest groups. These are folks and I know you are very interested in the Raspberry PI, and I'm sure we'll get to that. But these are groups that come together and say, Hey, we want to enhance this in some way. To add extra stuff to it, extra functionality like Red Hat Enterprise Linux and Core Rocky Linux will not boot on the Raspberry PI, for example.

You can't just slap it on an image, it won't work. We have a group around making that happen. The packages and the things that we produce don't get put into the core like default Rocky Linux, which matches Red Hat, exactly. But we have other repositories and so forth that we can add our extras into. There's a whole lot of this stuff we have a desktop crew that likes to KDE or MATE Images, which are different desktop environments. You don't get that if you just install Stock Red Hat Enterprise Linux, you will not get KDE. It doesn't exist for it. They do things like throw live spins. We have a KDE live version where you can just grab the ISO and boots right to the desktop. Red Hat Enterprise Linux does not have that. That's not in core Rocky Linux, but it's part of a special interest group or SIGs we call them.

David Rush:

I got to pull back for just a second here and point out. It's mostly guys here, but I did see TD Washington here. We got a lot of old timers in here. Mostly everybody's enraptured because we usually have a lot more posts going on. They're all just sitting on the edge of their seats. But I did a presentation one time sometime last year about all the various desktops and tried to put some effort into convincing everybody that it is in fact MATE and not mate. Oh, thank you very much.

Skip Grube:

It's MATE, I still say mate in my mind a lot, but it comes from a tea in Argentina I think. Because that's where the fellows from, but it's weird.

David Rush:

Alright, well let me dive into to some of my questions here and we'll hopefully find some stuff that Greg wants to pile in on.

Greg Kurtzer:

Actually, can I jump in? Can I jump in real quick? First thing I got to ask Dave, what are you drinking? That looks awesome.

David Rush:

There is a trademark here. I wear a different Hawaiian shirt and I drink the same iced tea every show in the same cup. It's just iced tea with lemon and some keto friendly sweetener.

Greg Kurtzer:

We've had some Rocky parties in Zoom where we encourage people to bring their drinks. We don't record the conversations and whatnot. Just in case because people are drinking. But we get some pretty cool drinks and people even doing shot games.

Skip Grube: 

Rocky After Dark

David Rush:

Will do today is join the mailing list and see what level of participation I could do. One of the things that I learned about you, Greg, is with your long and glorious career of starting all kinds of organizations. I go back to CAOS. Were you involved in any way with White Box?

Building Rocky Linux [17:11]

Greg Kurtzer:

I remember White Box quite a bit. When we were building CAOS Linux, we were focused on a new distribution of Linux that was RPM based. We even, we even knew right from the beginning we wanted to use a, well, we architected the operating system a little bit differently. Not to go on a tangent, but real quick, because this is interesting. We architected the operating system a little bit differently. What we said was the core of it, we split the core, which we said had to be a completely stable ABI, completely stable base. But all of the packages around this core could change versions, they can update. That was how we looked at it. When we built this core, this core had to be completely self-hosting, completely reproducible, completely rebuildable.

It didn't have everything needed to boot, didn't have a kernel, it was the user space. Then every other package that we built from there was built inside of this core. Every dependency in the chain had to be specific for each package coming into it. But that core was based on Red Hat way back then. This is before Red Hat Linux was end of life. This was before Red Hat Enterprise Linux really took off. Now a little while Red Hat ran Red Hat Linux, which was freely available. Then Red Hat Enterprise Linux, they started for some reason with version two. I don't know why, but right afterwards they came out with version three. I got and reason why, I'll talk about that in a moment.

I want to hear that. We were building CAOS Linux, which again was based on Red Hat Linux at the time. The first version was Red Hat Linux. Actually I may even be off on that. This was a very long time ago, this was 2002, 2003. But that's what we were building on. Then Red Hat end of Life, Red Hat Linux, and basically said, if you want to continue using Red Hat, you have to go use Red Hat Enterprise Linux. That obviously affected what we were doing and it affected a number of people. We were still talking about this and trying to figure out what's the best bet for us when all of a sudden there was a slash doc post of White Box Enterprise Lins coming into the world.

The first rebuild of Red Hat that I know of was in fact White Box Enterprise Linux. For some reason, I'm forgetting the person's name who was running it, but he worked at a library, a parish library, and I don't remember exactly where it was now. But Oh, very cool. Yeah. He came out of a library and hosted this out of a library using the library's resources. It took off to such magnitude that slash dot hit it. Back in the day we used to talk about things getting slashed dotted, and White Box got slashed dotted so badly that nobody could even see what a project was. All the servers were down because they couldn't keep up.

We talked about it and I reached out to the maintainer and I said, does it make sense for us to work together? This is an interesting project, here's what we're doing and whatnot. I didn't get a response from him for quite some time, probably due to just the fact of how much he was dealing with. Then after a little while, he basically said I'm doing this for us and for the library. I didn't anticipate this being a giant, massive thing that where I'm maintaining this for the whole world. If you want to do something here you got my thumbs up. We decided we already have build infrastructure. We already have developers and engineers let's go and start building this. The first person who started working on this was a gentleman named Rocky McGough, and Rocky, he was deep into high performance computing and high performance clusters, HPC. He was already doing.

David Rush:

Sorry, but I know how close you and Rocky were. There's a lot of press about that, and I am sorry for your loss and for Linux's loss.

Greg Kurtzer:

He was a huge fan of open source. Everybody that knew him knew he just, he loved the community. In person he was a shy guy but he was brilliant, loved open source. He basically said, I've been doing this already for the community, for my company, for the community for some time. We had a few other people that were also involved in the project and were doing that previously for their companies. He basically said, so I can start this if you want and we can start rebuilding this, and we can leverage then those packages for CAOS Linux as well, those core packages. We were like, that's fantastic. Didn't even have a name for the project. We just called it CAOS EL for Enterprise Linux.

He started working on version three and we, he was going about it, and he heads down just chugging away at this. He gets to the point where he is about 99% done. I refreshed myself on some of these old mailing list posts from the, from the archive. Just so I can remember what happened in all of this. He was about 99% done. We announced the name change to CentOS. CAOS EL became CentOS, I was leading the project and we continued it. But at some point we get this weird message in IRC, somebody joins IRC didn't recognize the name, and she self-identified herself as Rocky's girlfriend, and to tell us that Rocky has passed away.

It was incredibly, as you can probably imagine, awkward and weird aspect. Especially considering he was just about done. We had some other people take over. Guy named Lance Davis took over for version three. John Newbegin was doing version two at the time which started after version three. We were able to get everything released and get it out there, but Rocky never got to see what was to become of his initial work and where CentOS would go.

David Rush:

Tragedy.

Greg Kurtzer:

Knowing him, that would've been just amazing for him. He would've loved that. In the early days of Rocky Linux, before we had a name, somebody asked me, they said, what happened with CentOS? Like, how did it start? I was telling this story as we were talking, and all of a sudden it just hit me. It was like, that's it, that's the name of the project,. It's meant as a tribute in honor of what he did, but also the fact that he never got to see it. That always broke my heart. I wanted to dedicate it to him. Every single person that heard the story there seconded it and was like, yes, that's it. Then every person since then always said can we rename the project? Oh, and we don't like the name of the, because they didn't hear the backstory of it. They didn't know why were we calling it Rocky. For the first six months to a year, Skip, how many times did we hear the same thing?

Skip Grube:

We still get it sometimes. Very occasionally somebody be like, well, I like your Linux distro, but the name's really weird. You guys ever think about changing that? No.

David Rush:

That's so odd. You guys wrote about that or answered it in an interview and it was addressed right off the bat. I would think that's as common now.

Skip Grube:

People wander in. Not everyone's the most informed and that's okay.

Greg Kurtzer:

Skip for a while we even had it on the front page of Rocky Linux.org. Even the message that I sent.

Skip Grube:

It was funny because I remember this moment and what I was talking about in the Slack channel that we had joined, and this was like a couple days in and there was still chos because it was a million people all talking at once. I think we had a whole channel dedicated to what do we want to name the project or something? Maybe it was just general, but there were so many suggestions. I had a suggestion that I can't even remember what it was, but I'm pretty sure it was stupid. Finally Greg's like, I knew this guy who was original basically bootstrapped original CentOS and he passed away. It was funny because you get 5,000 voices and it's very rare that you can get them to all agree, but literally everyone was like, that's the name. It was a moment there and we were just like that's going to be the name.

Where CentOS Got Its Name [27:03]

David Rush:

Not to detract from that, but let me go backward for a second. Because I never did read anything about where CentOS got its name.

Greg Kurtzer:

That name was coined by, I mentioned, Lance Davis. He came up with the name and he proposed it to me. We went back and forth on this for a little bit. I said I really like the name CentOS, but I really hate separating the OS from Cent. I don't like CentOS. But, CentOS doesn't matter. But the reason why is, at this point in my life, I was incredibly pedantic and annoying and whatnot. A cent is cheap, but it's infinitely more expensive than free, which is what it was intended to be. Nice if you separate out the OS and you have now OS and you have Cent and it means something. I said, that's not what it is. It is community enterprise operating system. just say the whole thing through. however it turned out to be, I don't know, we went through a bunch of logos, but the logo that was prominent was Cent and with OS, and from that point on name was a mixed mash of CentOS, CentOS and Cent OS. For a little while there I had some grumpiness around that, but then I got more mature and grew up a little bit, just a little bit.

Upstream Vs. Downstream Distro [28:37]

David Rush:

CentOS is an upstream distro, Rocky is a downstream. Can you tell us what that means?

Greg Kurtzer:

I'll take the first bit and then I'm going to hand it over to Skip to fix everything I said that's wrong,. CentOS started off in the early days. This might be important later in our discussions, I'll mention this CentOS was always a very small team. It was a small team because of the security implications of what it was we were doing and how to manage that. Now we call it the supply chain security of each one of those packages, but back then it was just, how do we manage the private keys. We can't give everybody the private keys, and people were building it literally on their own systems, their own workstations, their own desktops and whatnot. We had to manage the control structure of security and the easiest way of doing that is just to keep your circle of trust very small. CentOS, in the early days it was a very small number of people and that persisted for a very long period of time. As I completely went down that track, I forgot your question, Dave. Remind me your question again. Oh where is it? Where is it? Okay, yeah, I'm good. Okay. I need more coffee.

When we first started doing this, we were literally logged into and we purchased Red Hat Network subscriptions, and we were downloading the software piece by piece from the Red Hat Network. Now, a little bit later, we were able to also download it from an FTP server that we were able to get access to. But initially it came right out of Red Hat Enterprise Linux, like there was nothing else. At that point, Fedora didn't even exist. This was just Red Hat Enterprise Linux, Red Hat engineers and developers created those packages, did all the patching, put it all together, and then release the source RPMs. Either via FTP or the Red Hat Network. We would log into that and download these bits of source code, the patches, the source, RPMs, the specs, everything that we needed to go and start the build process.

To Skip's point, it's never as easy as just going RPM rebuild on the source RPM. If you're going for binary output that is mirroring another binary, you have to mirror that entire build infrastructure which is difficult. Skip, I'm sure talk about this here in just a little bit more. CentOS was always downstream. It was always Red Hat Enterprise. Linux was in front, CentOS was below, later Fedora came along and Fedora came out highly in front, very far in front. Then CentOS, excuse me, and then Red Hat, and then CentOS now. Skip, correct me if I'm wrong on any of these dates or approximate dates relative. When CentOS 8 was released, there was also something released called CentOS Stream, which gave the ability to see into newer versions of packages.

You can install CentOS and you can activate Stream as a repository. Now what the announcement was that we just showed the blog post was that CentOS Linux is going away, but CentOS Stream is continuing in taking over and it's going to be in front of Red Head Enterprise Linux, or more pedantically between Fedora and REL, but much closer to REL. The goal of it as it was initially advertised, and somebody's going to comment on this, and I know it because it happens all the time. But it was originally messaged to the world as a rolling release. It's always just perpetually updated and you can go grab a snapshot, but they are going to do snapshots from installers and whatnot. But it is a rolling release.

I've also heard people at Red Hat call it a beta. I've heard people at Red Hat call it, where they're going with it, like preview release. There's been a lot of names for it and there's been a lot of confusion in the messaging for it. But all in all, what it is it is a community. I'm going to use this word also loosely, look ahead of what's coming. It gives us the ability now to sync into stream and to see what's coming into Red Hat Enterprise Linux from Stream and the same tags. Again, Skip, I'm expecting to correct me, the same tags that REL uses are represented in the stream Git repository. We can pull from the same exact tags that Red Hat Enterprise Linux is pulling from, and then use that as our mirror source. It went from behind REL to now in front of REL. Interesting like that. Yep. How did I do on that?

Skip Grube:

You were close 9 out of 10.That's the gist of it really. I think that it's important to distinguish because a lot of people get confused. Again, the naming of the naming is weird. I have no right to talk about naming though I've named things terrible in the past. CentOS and CentOS stream, they are similar, but they are different things, they are not the same. CentOS Stream, as Greg said, is a Red Hat Enterprise preview. The versions in CentOS Stream will be in REL in about a year's time. We're talking six to 18 months, somewhere in there, depending on the time of year. He's quite right when he says that it's a rolling release within major versions.

I'm talking about CentOS Stream now. CentOS Stream 8, there is no CentOS Stream 8.1, 8.2, 8.3, et cetera. The packages just roll in. We updates happen organically almost. Okay. But CentOS 8 Stream does not roll into CentOS 9 Stream. That's where the confusion comes in because when people think of rolling distribution, they think it's like Arch Linux. Arch Linux correct me if I'm wrong, but I'm 99% sure Arch Linux is a true rolling release distro. They have Arch Linux and there's no Arch Linux 10 or Arch Linux 11 or whatever. It's just packages and they continuously get updated according to the dev team schedule. but CentOS Stream, not like that.

CentOS Stream 8 is rolling release within the bounds of 8, and CentOS Stream 9 is rolling release within the bounds of 9, but they do not cross over each other. CentOS Stream 8 does not roll into 9. It doesn't work like that. I would say rolling release within a major version, half rolling release, if you can think about it that way. Whereas Red Hat Enterprise Linux and Rocky Linux, the rebuild of it, is very segmented. We have Rocky Linux, 9.0, 9.1, there will be a 9.2, a 9.3, et cetera. They still roll right. If I'm sitting at my console and I do a DNF update, which is our update command I'm on 9.0, our latest right now is 9.1, I will get 9.1 automatically.

David Rush:

Do a DNF DIS intro or is it just DNF update?

Skip Grube:

There's no special thing. There are updates even within 9.1 for example. 9.1 in say November is going to look a little different from 9.1 in December, although there's not major stuff does not change within what we call a minor release. 9 would be the major release. That .1 is the minor release. Red Hat who is our upstream provider introduces new features and new things in minor releases. 9.2 will have some new stuff that may not be quite as compatible with all the stuff in 9.1, but you can still update to it no problem. Hopefully that makes sense.

David Rush:

A lot of sense. That's a lot of things for me. I appreciate that guys.

Skip Grube:

I think the important thing to take away from this, and part of the reason for Rocky Linux's existence is that lots of people, and I think Red Hat perhaps underestimated how many people think that CentOS is awesome. They love it. When it was announced that, oh we're going to discontinue CentOS and just go with CentOS Stream, it's very similar for sure, but it's not the same. A lot of people I think really desired the same. Plus there was the whole thing where..

David Rush:

Huge within the industry.

Skip Grube:

There was the whole thing where like, well we have we cut off eight or nine years of updates there with the discontinuation in 2021. That was a little odd. It was like, well, maybe we started in 2019, maybe we shouldn't have even started it, if we're just going to cut it off in 2021. But anyway, but basically lots of people came together and was like we should do something. We did, hence Rocky Linux, we try to be the spiritual successor anyway.

Raspberry Pi [38:59]

David Rush:

Alright, so I'm going to dive right into the core of what we do here and maybe get it out of the way. I was casting about one day doing some research or other and ran across a blurb that says, Rocky available for RAS Pie. All projects get put aside. I have a regular job and much consternation to my boss as I disappeared down the well for a couple weeks and didn't do anything really productive for my company, but it just had to be played with. Rocky 8 on RAS PI. I do regular DNF updates on it, but I was stunned. I was shocked. You're not the first major distro to get, I'm going to use the word ported. I'm not a coder or developer, sadly, but we'll call it ported for the PI. But you were certainly, to my knowledge, the first I'm going to get in trouble here. You're changing my pronunciations of things with REL have called it R-hell all my life and I'm going to have a tough time changing that. You are, to my knowledge, the first R-hell downstream or upstream distro that went on the Pi. it just gave me a whole new world to play with in large part for Linux plus courses and LPI courses that I teach. Hey, you don't have to put this in a VM anymore. You spent your 35 bucks for your Ras Pi. Now, you can have live. Tell us a little bit about that if you could.

Skip Grube:

I've honestly, I think I've heard R-hell before, but I don't know, that sounds like ominous. Oh, I got to go fix R-hell again.

David Rush:

Used it for exactly that reason.

Skip Grube:

I come from a different tradition. Honestly, that post you saw about the Raspberry PI port, it might have been mine actually. This came from almost a realization and a desire to scratch my own itch, quite frankly, as far as the Raspberry PI goes. I also like the Raspberry Pi a lot. I've got a couple here, well one's downstairs I know. I like self-hosting a bit. I don't go crazy with it, but it acts as my home server. I've got like a media server on it and it holds my files. I even run classic Doom. I love Doom Game Server.

David Rush:

Your presentation on how to install and play Doom on R Pi.

Skip Grube:

I run a Zandronum Multiplayer Doom server on it. I was thinking, so I had a Ubuntu server on this thing before I joined the Rocky Linux Project. This is actually part of why I joined, quite frankly, because it was always my ambition to upgrade and stick CentOS on it. Part of it is laziness because I'm like, oh, well CentOS 8, if I can upgrade to that on my Raspberry PI, then I'll be set for 10 years. I don't have to go and like swap all my stuff out. I'll just DNF update my way into and it'll just run itself. When the CentOS discontinuation announcement happened, I was like, oh no, my dreams.

What happened? After getting into Rocky Linux proper, because I do a lot of what we call the core work. Which is the bootstrapping new versions, troubleshooting packages, that thing. I consider myself a junior member of the dead team quite frankly, but we'll get into that later. I noticed that there is a guy, and I wish he's part of the CentOS community and I wish he was here, but he can't be here right now. His name is Pablo Greco, and he's amazing guy. What he does is one of the many things he is an ARM enthusiast, ARM processor enthusiast. Big shout out to him, by the way, he's a wonderful community member. One of the things he does is he maintains a kernel port to the Raspberry Pi because for several technical reasons, you can't take a stock Red Hat Enterprise Linux kernel, compile it for the ARM processor and boot the Pi with it.

It doesn't work without major modifications. It has to do with firmware, there's drivers that are missing. With a lot of work you can get it to come up and we're still looking into this, but it will be missing like Wi-Fi for example. You won't have that or the graphics drivers and that thing. Pablo he really does, in my opinion, the heavy lifting, the core work because I'm no kernel developer. I do packaging work but I do not consider myself. I'll compile kernels sure. I fiddled with their settings sometimes, but I do not have, in my opinion, nearly the deep knowledge to do actual kernel development, but Pablo does. I asked him a little bit about this and was like, oh my gosh, gosh, you have this kernel that works with Red Hat Enterprise Linux and Rocky Linux, and it just works with the Pi.

We can develop a package for it. If we combine this kernel and the rest of the packages we already compile for ARM 64, like out of the box. We already have a Rocky Linux ARCH, we call it Arch 64, but it's ARM 64 version of everything. We do that by default. I was like, so if we take your kernel and the ARM 64 packages and put them together, we can make an image. Really the only contribution that I really make is I create the images to make them easy for people to use because Pablo's brilliant and it was possible to take his stuff on his he has a web folder where he throws a lot of his work in and it is a gold mine there.

It's amazing. I love him for it. but it's not the, you have to figure some things out yourself. There's not really an easy I would say definitely doable for sure, but it's for a lot of folks that you just want an image file that you can write to your Raspberry Pi and it's official more or less and it's ready to go. I was like, well I can, and I did this almost like selfish desire to. I was like, I want Rocky Linux 8 on my Raspberry Pi so that I don't have to switch over and like, I don't have to upgrade a Ubuntu server in like three years or whatever. Because I don't want to like, so and that's, I got into it and I got a little further into it than maybe I expected to at first. Really what I do is I take the work of others and I make it presentable. Which is what I don't want to be like, I did all this effort it's not really, in my opinion, it's not that much effort. It's more of a last, have you've heard of the last 10% problem where all the work's done, but to make it not perfect, but to make it usable for most people, there was an extra like polishing going on there.

Greg Kurtzer:

Skip, that's the role of the distribution. I mean all of the sources that we're leveraging, I mean the amount of new code or even edited code that a Linux distribution would implement in terms of the overall code is incredibly small. What we're doing is we're leveraging the entire community, I mean, not just like the Linux kernel and you start going through every single package and all of the people that have been part of each one of those packages, both the development contributions, bug reports, I mean the picture is so big. The Linux distribution to Skip's point is really, we are just there assembling that final mile so it could be consumable easily and leveraged easily. This is one of the reasons why, honestly, I truly believe that Linux distribution should always be freely available, is because all of those upstream sources were released to the community with the intention of them being up freely available. I personally do not like stopping that or interrupting that from the end users, right at that final last mile, that final stage. To me, I do believe that the culmination of all of the work from the open source community does need to be freely available to everybody because that was the intent that every piece of every package involved in that distribution, that was the intent of every package in there.

David Rush:

Couldn't agree with you more. Okay, let me go down two roads here. First of all, kudos and compliments to the image and the distro. Everything that I have ever done with it works perfectly the first time out. If there's a package designed to run on an X86 based flavor, there may be exceptions. I haven't done everything yet. But I have been running for a year and a half, two years Cockpit on there. That was our management console of choice when we came to teach that. I didn't have to do anything. I didn't have to do any configuration changes to make it work or any recompile. It was plug and play and now all we got to do is learn the product. It's been that way with every project that I have ever done. I've got a little bug here and I got a little challenge there, but it works. Awesome. Thank you guys. I'm sorry, go ahead. Skip.

Skip Grube:

I said we stand on the shoulders of giants. Those giants also stand on their shoulders. It's giants all the way down.

David Rush:

An enterprise such as the ones that you guys have been participating with for so long, they're not cheap. I'm thinking I can't take the money that I made last week and start a new distro and have it turned into Rocky. How do you do that?

Greg Kurtzer:

How actually you can, and here's why. There was a number of tenants of what we created Rocky and the Rocky Enterprise Software Foundation to do. It was very important to us that what happened just to be blunt, and I don't mean anything ill out of any of this, but what happened with CentOS doesn't happen again. We wanted to make sure that we're creating a structure, an organizational structure that the organization is protected in as many ways as we possibly can. But even more than that, we have to create the technology such that somebody else can pick up from wherever we are, wherever we leave off, and go and continue this and continue running with the torch. Or create another option. One of the first things that we identified, so let me back up a little bit.

When we first started this process, Skip mentioned that we created a bunch of different channels and he's not joking with regards to how much velocity of communication was happening on these. Just to demonstrate it, we were using a free Slack tier. You get 10,000 messages in your free Slack tier. If I didn't respond to people in a couple days, messages would poof because we had so many messages coming in, and it wasn't just that they scroll off the screen. Like I had direct messages to me that I'm like, okay, I see that there, I'm going to get to that one later. Okay, next day goes by, and if I went like two or three days, I see that there's still a direct message from there and I click on it and it's empty and it says go buy Slack to see this message. It became very clear and we had very quickly, we had over 10,000 people join. It was about a month and a half. That's how quick that the community took off from there. It's been just a remarkable experience. I don't know where I was going with that.

Rocky Enterprise Software Foundation [52:01]

David Rush:

Well, and I'll bring you back around then. You mentioned in our last end of topic the Rocky Enterprise Software Foundation.

Greg Kurtzer:

I need more coffee. I didn't sleep enough last night. Yeah. Keep me, keep me on focus you guys. Wow. I need help. One of the things that we identified is the thing that's most important of what we're creating is not just the operating system. It is everything we need to do to build that operating system. We started actually not building operating system. The primary role of what we started doing collectively as a group was to start with the infrastructure. Now Skip immediately identified, even while you're doing that, I'm going to focus on figuring out things like package build orders, dependencies, hidden dependencies, things that are not in the operating system but are dependencies to build and to get binary compatibility. We had to go start figuring out all of that.

Skip was working on that, but a large lion’s share of the organization was working on building that build infrastructure such that we can have members of the community come and be part, at least that was the goal. Now, we started off using Kogi. Kogi is a fantastic build system that Fedora uses and Fedora maintains, and it is tremendous. It's awesome. But one of the things that we notice is it's built for predominantly physical clusters. Somebody who has a physical on-prem or hosted cluster, a resource racks of servers that they can throw builds at and manage those builds in parallel and whatnot. It's really good at that. But it's not so good at running in the cloud. As we mentioned before or at least in the green room, we've had so much support from cloud's providers that we wanted to make sure we're leveraging these cloud providers to build the operating system.

We build it in a secure manner in which other people can come in and join and be part of this big project. We identified right away that we were going to need to build a build system, so we did that. We've now released it. As a matter of fact, we didn't even want to release version 9 until we were ready to release the build system with it. The moment somebody goes and downloads any version of Rocky, you can actually also go and get all of the infrastructure necessary to go and rebuild Rocky yourself. To your point that you just mentioned, it's really important for the longevity of not only the project but of Enterprise Linux itself for Enterprises, IT organizations, and people, engineers, system administrators, to have so much confidence in what is Enterprise Linux. We're doing these air quotes a lot today, Skip. What is Enterprise Linux and know that there's always going to be a suitable enterprise Linux solution that provides that level of compatibility, pound for pound, as Skip said with the standard.

RESF Support Resources [55:14]

David Rush:

So RESF is a support foundation that provides resources for all of this development to happen?

Greg Kurtzer:

I'm going to be very pedantic here because I want to make sure that I get this correct, and everybody understands very well. When I posted that message to the blog, I was actually talking about CIQ, so I was thinking CIQ is going to come and go build this, and we're going to go make this a project with the community. When I saw 10,000 people joining that community, it was clear, this is not going to be a CIQ effort. It can't be a CIQ effort. This has to be a community effort. We have to do the community in such a way that we're going to protect the project first and foremost and then create a foundation that not only supports the notion and ethics and morals of what it means to be open source, but at the same because there's already foundations that do that. But there aren't foundations that really also cater simultaneously to the focus of where we need it, the users.

In our case, the users are predominantly professional system administrators, IT organizations, and large enterprise organizations. That is our community and that is where we need to be really thinking about and making sure that, yes, we're abiding to the morals and standards of what it means to be open source, but we're also providing something that really cross pollinates the needs and the necessity of what enterprise organizations require. That was the guiding light of what was RESF to be and how are we going to get there. RESF, it stands for the Rocky Enterprise Software Foundation. It is a public benefits corporation, and it is a separate entity and it runs itself, it is managed by the community and it is a standalone entity.

Now, CIQ, just to paint this picture being a for-profit company does not control and does not hold hostage or even own anything about RESF. Aside from the fact that CIQ contributes literally now millions of dollars to the RESF to help enable this and help enable what we're building and what we're doing. Aside from that there's no sort of control structure. As a matter of fact we've built in control structures to the RESF to ensure that companies, including CIQ, cannot hold the project hostage in a negative way. From a business perspective, that means CIQ has to add value, true value. Not by again, holding it hostage and saying if you need this, if you want this, you have to pay us. No, we're never going to do that.

We can't do that. Instead, we're here to add that value. It forces an ethical business model as well. That's kind of, sorry, I don't know what that sound was. That's where we came to be with regards to the Rocky Enterprise Software Foundation. It is at standalone entity and that's why lots of individuals as well as organizations have supported this. If you look at our sponsors and you look at the companies that are part of what it is that we're doing, you're going to see a very impressive lineup of organizations.

David Rush:

You've got some ex RESF has some exciting news underway right now. Yeah.

Greg Kurtzer:

Oh yes, absolutely. Within the last couple months we've gotten all of our new bylaws written, our new charter. We've had it validated by various open source experts, legals, Heather Meeker, if anyone knows of Heather Meeker. She's now validated it, looked over it and, and contributed to it. Again, what we've created is this foundation that caters to not only open source, but also our user base, which is enterprise and professional IT organizations and people. All of those bylaws are now done. We are right now in the process of a vote on board members. Right now, Skip nor I know who's going to be on the board starting in January 1, but we have voted and we're waiting for all those votes to come in.

We are anticipating by mid next week, we will know who's going to be on the board and we're going to be having a press release shortly there afterwards. The board is comprised of individual members of the foundation, so contributors like Skip and myself and others, as well as several independent at large directors. These are people that are not associated with the project, but do bring value into the project, either as users or contributors or open source people and other project leads. All of that's coming together right now. The one thing I really want to stress as we're talking about board members is, even though we have sponsors to the RESF, we will never sell board seats. In my mind, this is a sure fire way for organizations to come in and start manipulating the agenda of what an open source foundation is doing. We never sell board seats. That's a very important and critical distinction in our mind.

Rocky Linux to Distribution [1:01:03]

David Rush:

We've certainly seen Holy Wars on either side of that coin so kudos to you. Right after you posted in the chat feed, somebody posted a question. I'm not sure who's going to best answer this, so we'll just throw it out. Tullowit who's a regular here, and he is a moderator on the unofficial Discord channel that supports our efforts here. He wants to know what does it take to go from, let's add this feature to Rocky Linux to making it part of the distro. If somebody's looking at the stream release perhaps, or you're looking at the most recent binaries that are final with REL, I can do that. There's a new feature that somebody looked at and says, we really ought to incorporate this. How long from that germ of an idea to delivery? I'm sure it's not answer.

Greg Kurtzer:

I'll take the first stab and then Skip already talked a little bit about sig, so I want Skip to come in as well. We've had a lot of requests for the upstream kernel as an alternative kernel to Rocky Linux. This has been a lot of requests, like not a few. I mean, this is a lot of organizations. Usually it's the larger organizations that are either interested in the security model of the upstream kernel or some of the performance benefits of the upstream kernel. Rocky and Rocky Stream, huh?

I don't know if I'd call it Stream, but yeah. But the way that these come to be is through the special interest groups that Skip mentioned earlier. If somebody has an idea, they have another package they want to include, they have other optimizations or changes.

I'll put out a little poke. If somebody wanted to have a system five enabled operating system, because that one comes up a number of times, it makes me nervous to, Skip. But with that said, I'm still a system five knit person, so I still go back to that. If somebody wanted to do that, they could, they would have to create a special interest group. They'd have to design a charter for that special interest group and set up the appropriate leadership structure with that special interest group. If you just have a package, a single package that you want to bring in, and a special interest group does not currently exist that would fit for that package then we would, we would talk about creating a special interest group that would be best for that package.

Generally speaking, the intention here and the build system that we created called Peridot. Mustafa, if you see him in our matter most, which by the way, we moved from Slack to Mattermost because of the problem that we mentioned regarding message retention among other things. We move to Mattermost if you do see Mustafa and Skip as well as Neil, Lewis, and others on the team, they're very familiar with the Build system. If you do have any questions on that, please do feel free to ask. That's a little teaser on joining our getting everybody to join our Mattermost. But with all that being said, the build system is designed to support special interest groups and additional packages. If you come into the Mattermost, you come into our chat or IRC and you want to start maintaining something special. Some addition to the operating system, we can do that through the special interest groups initiatives and structure that we already have. That's how we would do that. Skip, is there anything that I wanted that you wanted to add on a technical side or other.

Skip Grube:

Not so much technical, but it's an amazing question and we actually get it a lot. I think one of the, I always say this in our release meetings, but build it and they will come. I'll give you a good example is how the Raspberry Pi image got started. We had a chat channel in our, and by the way, Greg talking about Mattermost, I would say chat.rockylinux.org and we have bridged IRC channels with it for folks who like IRC. Is the nerve center of Rocky Linux development and conversation and everything.

David Rush:

I just posted the link to the Rocky website and it's in the tabs right on, I think not in about, but under support.

Skip Grube:

We have all kinds of folks there for all kinds of reasons. Lots of troubleshooting happens there, lots of development discussion, just everything people are interested in. It's organized into separate channels and so forth. We had and we currently have an altar channel alternative architecture. Which the Raspberry Pi qualifies because it's not supported by Stock Red Hat Enterprise Linux. I mentioned before I was doing this work but I didn't really have it. I was chatting up with folks on the chat channel and asking for ideas, advice when I'm trying to solve a problem or something. But the original I recommend doing iterative development. Try something.

You can compile things yourself, no problem. I’d do it yourself. I'm all about do it yourself. With help and advice from other people, obviously you want to collaborate and get something going and have a prototype or something to show off and be like, Hey, look, we would love here's the RPM that we want to include in a special SIG repository or something. We want to develop it further, and then we can like the Raspberry PI image started out on my home server. Then was distributed to friends for testing, and they were like, this is awesome. Let's publish it. Then eventually we work our way up, now it's on… if you go to the downloads page and there's an alternative images section, and you can pull it right there. I encourage not taking the cart before the horse. Don't set up this entire giant thing where you're going to have all this stuff going. Let's go product, and then we'll go we'll formalize it more. I don't know if that's business speak, but it's what they call that product driven development. I don't even know.

Prioritizing What to Add [1:07:57]

David Rush:

Looks like that answers another question that came up, but I guess there's a half question here. We can still hit, Andre posted at two minutes after the hour. How do you decide what to add and what to skip? When does an idea come out where everybody says, no, no, no, no, no, not that.

Skip Grube:

I'm going to give you some more examples here. First of all, I did, I put a little blurb in there about it, but so Core Rocky Linux, forgetting the SIGs for a moment, the special interest groups. Core Rocky Linux, it's actually very easy to decide what to add, what not to. What is Red Hat Enterprise Linux doing? That's our whole mission of Core Rocky Linux is to match it, again, pound for pound version for version everything. If someone wants to create a newer version of a package. Something that came up the other day actually is someone said, Hey, I don't like the version of USB Guard that's included in Rocky Linux. Unfortunately, our answer to that is, well we cannot put the new version of USB Guard into Rocky Linux, because our goal is to match 

Red Hat Enterprise Linux [1:09:11]

However, if you would like to, you can totally grab the upstream source from Fedora, compile it against Rocky Linux Stein, there might be some work involved. I'm sorry to say, there's effort involved in everything. You can get it working, and if you get it working for yourself, you can share it with people, and if people are interested, we'll host it for you. Absolutely. We'll put it up on our gitch. We'll even make it available in a repository space for you. It won't be the primary Rocky Linux repository. People will have to enable it. It's an add-on. Again, there's steps towards doing that, and there's a lot of information in the Wiki about if you want to found a SIG.

We also tell a lot of folks that if you want a one-off package like that, Rocky Linux might be appropriate. But also there's an am there are amazing third party repositories for Red Hat Enterprise Linux, and one of the most popular is called EPEL. I don't know if you've heard of it, but it's Extra packages for Enterprise Linux. Especially if you just want a slightly newer version of something, or if you want this one package and not it's not a big thing like the Raspberry Pi support, it's just a single thing, then it might be appropriate for you to head over to Fedora and check out EPEL and see if you can get hosted. If you can become an EPEL maintainer there, highly recommend checking it out. They're not really upstream from us, I suppose, but they're definitely a great community there that maintains a lot of stuff. We have conversations about where stuff fits. There's a million situations and there's a different answer for each situation.

David Rush:

We've got a really wide variety of audience, and most of them aren't here live. We got a pretty good turnout for our usual, but by the end of the week, our viewership will go up 50, 60 times of what the live show is. Number of them are interested in pursuing development careers, though they're not always sure what that means. I've got a degree in tech or something like that, so I can code. Now, what do I do with that? Here's one of the things you can do. Take those skills and find a project that you like, find an application that you want to help develop, find some flavor of Linux that you like, do some bug research, maybe do some app development, whatever you've got. Here's a wonderful opportunity, and that's something I wanted to cover into this.

I side tracked quickly and easily, but there was definitely something I wanted to mention, When I was asking Greg about his, some earlier projects, particularly CAOS. I went back and looked at a lot of the old boards and discussions and things like that, and one of the things that stood out the most is how friendly everything that you've done is to all levels of contributors. From the most base newbie who only wants to contribute by saying I really like this, or, I'd like to try this, and I'll let you know what I think. All the way to a Skip or a Pablo level contributor or a Rocky level developer. Kudos to you on all of those accounts.

Greg Kurtzer:

The culture of the open source project is really important. I've learned a lot, and I obviously continue to learn a lot on how best to maintain that. Sometimes it's easy to get into arguments and whatnot, and we try not to. We've had a couple of reminders of how important it is to make sure that we're always staying on a positive vibe and a positive note. Again, there have been mistakes along the way, but mostly everyone on the team has learned from everyone of those mistakes, including myself. Our goal is to just do better all the time just to do better.

EPEL [1:13:36]

Skip brought up EPEL. If you don't mind, I'm going to share. I hope I'm not pivoting the conversation, but these are our update metrics, and it provides a really cool glimpse into what's happening with regards to all of the different enterprise Linux variants. This is data taken out of EPEL based on what is shared publicly with the world. As people connect in, obviously there's logs, they anonymize those logs and then publish a derivative of those logs so people can run metrics like this. This is about when the end of life announcement of CentOS hit right at the beginning of this graph. You can see it took us a little while to go and build Rocky Linux, and then to release something. It took us almost seven months to get that out there. It was about four months building the infrastructure, or predominantly focused on building the infrastructure, and then about two months taking the work of Skip and others, and then throwing into that infrastructure and then ending up with a, with an operating system in about another month of testing.

Then that's how we come to the seven months. You can see from right there, we started increasing. You can see where we've been going and what the trend line looks like. It's definitely the steepest acceleration out of all of the Enterprise Linux focused operating systems. This particular chart is looking at something called it's non ephemera. Usually I start with Ephemeral, but this is the non ephemeral, and ephemeral is short-lived systems. Systems that have just been provisioned and running for less than a week. We end up seeing a lot of CICD pipelines, a lot of microservices, DevOps stuff, testing stuff and whatnot ends up ephemeral. More of the persistent systems is what this graph is showing.

You can definitely see a really cool curve here. You've seen a couple times where Red Hat, CentOS and us have just traded back and forth a little bit in terms of leading, and it's been a fun race to watch. You can see the incline of this. If we've looked now at the ephemeral systems, we see a really different story, and it's a really interesting one as well. This is what you see when you look at the ephemeral systems. It's much more jagged. There's a lot of potential reasons for that. We don't have the specific answers on who is connecting, how they are connecting and whatnot. But you can see it is in fact very jaggy, but you can see it's almost like we've replaced the CICD usage of what Red Hat was, and we're taking over a lot more of that now.

And you can also see in both of these, the effect on Red Hat by end of life, I CentOS in terms of the increase of Red Hat Enterprise Linux. A lot of people ask why did Red Hat do that? Of course, we don't have the answer, we don't really know. But this graph tells a very interesting story. I'm sure that Red Hat at least I would guess that Red Hat was expecting some of that to occur in terms of uptake. They were very successful, I would think, in terms of that goal, if that was in fact their goal. But what I love here is just the amount of uptake that we've had and continue to have. Certain analysts that we've spoken with have shared that they fully expect this to continue. They're very much looking forward to what next year and the following year are going to look like with regards to Rocky uptake.

How Do We Characterize AlmaLinux? [1:17:38]

David Rush:

Well, so I want to work with this chart for one second. There was some news that surprised me. On December 7th, there was a news release out of Firmilabs and out of CERN, no, you're no doubt familiar. For everybody else, this lovely blue line down here is AlmaLinux. They started developing around the same time that Rocky started developing. Where do you fit them? Are they downstream? Are they upstream? Did they replace CentOS? How do we characterize them?

Greg Kurtzer:

They are basically solving the same area that we are focused on solving as well. They came out and announced that they were going to do something, I think a week after we did. I've talked with the founder. One of the reasons that he mentioned was he was trying to get ahold of me, but honestly it was very difficult at that time to get ahold of me. Because he couldn't get ahold of me, he decided to just go ahead and create their own and they did so. They announced it way back over here, and you can see they were out to market much sooner. It's hard to see when this line starts taking off right there, but it was quite a bit sooner than we were. They've done a lot of work to really get that out soon.

To give them a huge amount of credit. They're very fast in terms of releasing new versions, so point releases and major versions. We tend to hold back a little bit and it's nothing negative to what they're doing or their process. But we really want to make sure that we're going through all of the appropriate testing and making sure that everything is completely reproducible before we release. I would assume that they're doing similar, but I don't have transparency into that, and I would expect that they're doing great.

David Rush:

Of course the news which surprised me, just because I'm probably a bigger Rocky fan than am Alma, by far and away, is that both CERN and FermiLabs announced that going forward for the most part, and not a hundred percent their Linux distro of choice is going to be Alma. I wonder, did they come to you and say, Hey, if you give us this much more support, or if you do, then we will select you, or is that just something that they did on their own?

Greg Kurtzer:

I'm not privy to why CERN made the but I do have some I have gotten some feedback from the FermiLab side. From the CERN side, it seems as though they made some judgment calls that for whatever reasons they ended up choosing that. I don't have specifics into that, but I have a feeling from, based on feedback that I heard we did not release version 8 for PPC 64 for power. We did not release that for version 8. We wanted to make sure that we are getting the big mass amount of the community released first, which was X86 an ARM. We really focused on X86 an ARM. You can see even with that it took us a significant amount of time to get that released.

We didn't want to delay any further and really make sure to get PPC out there as well so we didn't do PPC. One of the bits of feedback that I've gotten was CERN does have some PPC use cases, and as a result that influenced their decision. Moving forward to version nine, we have not only PPC, but we've got S390 as well as X86 as you can imagine and ARM. We are working with a few people on Risk. We're hoping to get a Risk build done at some point. Now, it would be a very different type of build than what we're currently doing. It would be an alternative build because that is not supported upstream. We would be patching and making modifications and forking to do that.

Of course, offer all of our changes and whatnot back upstream if they're, if any of that is interesting to upstream. We're not sure exactly what the because was from for CERN to make that decision. But Fermi, who is basically Fermi does a lot of amazing research, but they work extremely closely with CERN and they're a major point of presence for a lot of the CERN computation and research that's happening here in the United States. As a result of that, they just followed CERNs decision point on that. That was the feedback that I got back from Firmi. But in either case, this is one of the benefits of having even multiple solutions within the Enterprise Linux family of distributions is, if anything ever happens to Alma, if anything ever happens to any of the distributions. Not to pick on Alma, but if anything happens to any of them, there is now choice.

This is part of why we want to make Rocky reproducible, is because we want to make sure that everybody who chooses to go with Enterprise Linux knows that they have the confidence that this is stable. Not just because one organization is stable, but because many organizations are doing the exact same thing. It is a very minor lift going from one to the other. One other point I'll just make on this real quick and then I'll stop blabbing at least about this is one of another aspect that Peridot changes. This is an area of focus that we've had really since day one. I spent a lot of my career working for the Department of Energy, working for US Federal. I know many of the aspects of what they need and what they're looking for.

Couple that with presidential mandate on SP supply chain security within the federal government, and validation of supply chain security. We've spent and are continuing until we release this we're putting together a huge amount of transparency. If somebody has a Rocky Linux system that they've built, they can go and validate a binary and say that binary came from that RPM. That RPM the check sums all matched, so I have confidence in that binary because it came from that RPM and that RPM is signed by the organization who distributed it. I have confidence in that RPM, but do you have confidence as we continue walking up the chain? Do you have confidence with the organization who built it? How do you know that you have confidence with them?

This goes right into absolute supply chain transparency and security. Peridot is adding additional functionality around not just SBOM logs and everything else that you need to get that supply chain security. But also cryptographic validation and signing of each one of those artifacts. You can go back and you can validate this binary came from this package, which came from this organization, which came from this build infrastructure, which came from this SBOM, and here's the entire build log of that. If somebody really wants, they can go back and follow that entire path all the way back up. That's something that is incredibly important to us. Then couple in additional security accreditations that we're doing. As an example of this, and I can probably turn this image off so I can see everybody.

As an example of that I mentioned how CIQ is putting capital and putting resources back into the Rocky community. We are funding a FIPS validation of Rocky. Anyone familiar with FIPS knows that it is not only a very long and drawn out process, but it is also a very expensive process. It's approximately $200,000 for each cryptographic module that you wish to validate. In the operating system, there's five major cryptographic subsystems that need validation. This is a million dollar investment and we are not only doing the FIPS validation for business reasons, but we're actually open sourcing all of that. Once it's done, we're giving that entire validation away to the community, including those binaries that have been validated to the community for free. Anybody can leverage that FIPS validation.

This is a really important step and it's a huge. For CIQ, now we're just past 70 people, so we've grown. We're getting to the point where we're no longer a small company, which is funny for me. Because I've been a CEO of a small company for a while now. We're getting to the point where we're now no longer a small company, but it's still a million dollars is still a huge investment. That's just for version 8. For version 9 is another million dollar investment. We're already starting to prepare for that now. You can expect, so 8.6 is going to have a FIPS validation on it, and 9.2 is going to have a FIPS validation on it, and all of that is going to be freely available again.

FOSS Support [1:27:44]

David Rush:

That's amazing. I'm chewing up a tremendous amount of your time here. You've answered all my questions and then some, even the ones I haven't asked you've taken care of. I got two quickies just to close out my list and then anything else you guys want to talk about or if you've had enough of me take off and enjoy the rest of the day. Okay. You are big FOSS proponents, as am I. I don't donate to a lot of organizations. Some I've picked up recently and my longest term one is Linux Foundation, to my mind, the largest FOSS supporter organization out there. Do you work with them? You just, do you benefit from them? Where does that fit? It's my own personal love.

Greg Kurtzer:

Yeah, that's a great question. I can say this and I'll stop after that. We have a very good relationship with the Linux Foundation, not only the foundation itself, but members of the foundation. I can give an example, Greg Kroah-Hartman, who is the maintainer of the stable kernel. When I was talking about bringing in the stable kernel support for Rocky as an alternative kernel, we are working with Greg directly on that kernel. Greg has been just fantastic. I can also mention that he has been nominated as one of the independent at large board members for the RESF. Hopefully we'll have some good news with regards to everybody who's been nominated, who actually got voted in. But he is definitely on the ballot. We are very excited and happy to be able to work with Greg on things like this. But again, the Linux Foundation has been just fantastic and tremendous, not only in terms of our partnership with them as a Foundation-Foundation partnership. But also just in terms of what they've done for the community. It's massive and we're very appreciative to everything that the Linux Foundation has done. I'm glad to hear that you are a donor to the LF.

CIQ Educational Program [1:29:50]

David Rush:

There's not much money in these pockets, but you got to pick some and that and EFF I love and a couple specific applications out there. Alright, last one. This one is all for Dave. What I do for a living purportedly is I teach, so I'm with an organization where we write books and make videos and our primary standup customer base are federal law enforcement agencies. We teach them what's a computer, what's in it, and enough background that they can address technology forensically. When they're on the stand, they've achieved some level of certification, typically using our materials or our standup instruction. What do you get in the world of education? What does CIQ have in the world of an educational program?

Greg Kurtzer:

It's a fantastic question. You mentioned LPI earlier. I have a weak spot for LPI. I mentioned, I worked for the Department of Energy, before I worked for the Department of Energy I worked for a little company called Linux Care. I don't know if you remember Linux Care back from the late nineties, early two thousands. That was one of the most fun companies that I ever worked with because I got to work with, I mean, just legends and just so many tremendous amazing people in open source. People that, just the moment they start talking, everyone shuts up and just listens because the amount of stuff that's just amazing that they can share. I'm not going to start name dropping everybody, but there's a lot of amazing people that came out of Linux Care.

I got to work there for a little while. It was fantastic. But the reason why I bring that up, because when I was there, the Linux Care had a fantastic role in some of these early Linux education programs. I believe Linux Care even wrote the first check of sponsorship to the LPI, so even way back then. I was consulted on a little bit on some of the early LPI documents. The same thing on the LSB side. They were a huge initial supporter of LSB. I helped out a little bit with the Linux file system hierarchy standard, and so stuff like that. I have a weak spot for all of that. Now to get to your question which by the way, I'm going to name drop actually one or two things since you remember Linux Care. Linux Care had three founders. Do you remember the three founders? Dave Sifry, Arthur Tyde was the CEO, and Dave LaDuke was the CMO. Arthur and Dave are currently with us at CIQ now.

David Rush:

Oh, that's wonderful.

Greg Kurtzer:

Getting the gang back together. CIQ has been a lot of fun as well. But to get to your question, The LPI has just done so much amazing stuff for the community, both in terms of the curriculums that they've created and the materials that they've created and whatnot. We have had some conversations already with the LPI in terms of helping and sponsoring there and being part of what the LPI is doing. I'm hoping we've had some really good success there. We're just going to continue pushing on that and helping everywhere we can within the open source community. LPI is a big one there.

Where Are You From? [1:33:47]

David Rush:

The last completely pointless unrelated question to anything at all. You got to love living in the internet age because of what we've just done here. You and I ran across each other, I ran across you in some thread on a Reddit subreddit somewhere. I was talking to one person from Alma and said, Hey, would you want to consider talking with us? That fell apart for very good reasons, but you were also in on that thread and Oh, wait a minute, this is a way better thing. Okay, let's reach out to you with this marvelously semi-anonymous messaging system that's part of Reddit. I said hey, I got this show, would you consider joining? Absolutely. Here's the name of my administrator to handle these things. Send her an email. I send Jessica an email and you guys all work together. I don't know Jessica, I don't know where she is. I don't know who you guys, I don't know where you are. She doesn't know who I am. She doesn't know where I'm at. But after a string of emails and a URL link, this is the internet age, I still don't know where you are. Where are you guys?

Skip Grube:

We talk about how we're downstream from Red Hat. I'm a little upstream from Red Hat. I am just north of Raleigh, North Carolina. The downtown Raleigh is about a 25 minute drive from my house.

David Rush:

Yes. From your job history, Carolina, yes.

Skip Grube:

NC State, go Wolfpack!

David Rush:

I was going with your work as an admin for the court system.

Skip Grube:

They come and pluck engineers like apples down here. They're like, oh, you can just join our company. Thank you for graduating.

David Rush:

And Greg?

Greg Kurtzer:

I spent most of my life in the Bay Area. CIQ even started off being a Silicon Valley startup even though pretty much everyone in the company is remote. I based it off of that and I've more recently, in the middle of Covid, I moved to Reno, so I am now in Nevada. Trying to get away. We're having some beautiful weather here, and I love snow, and I love actually having seasons. California's one season, which is just always nice. But I do like all of the different forms of weather and whatnot, and it's fun not to have earthquakes, not to have to think about it.

David Rush:

Isn't that far a drive from the Bay Area? You still get to go catch up with the old folks.

Greg Kurtzer:

That was one of the reasons where it was like we're close enough that it's just a few hours away. I do go back to the Bay quite often due to meetings and whatnot, and it is still, it's such a common meetup and location for so many people in our industry. It makes sense. I do go back and forth at a reasonable amount, but I'm, I spend most of my time here in Nevada.

David Rush:

Good. Well, is there anything that you guys would like to talk about or promote or anything like that while.

Greg Kurtzer:

Well, Dave, you got to tell us where you are now.

David Rush:

Oh, sorry. My regulars know, I'm in a northern suburb of Houston, Texas, a little tiny community called Spring. We went remote, my company right around the beginning of the Covid Lockdowns, February 2020. That sounds about right. While we do have some in-office activities still happen and everybody tries to turn up once every week or two, by and large, we've become one of those mostly remote companies. Go in for the occasional meeting. Our offices, I think, today exist for the physical video production that we do. We've got production facilities down there, and we do classes, we were doing hiring SMEs, we, and I guess we still. That's either bring them in for a week or two and have them shoot, or we've done some online over Zoom and similar sessions and turn those into. We're like so many companies now. I've got some very good friends that I've developed in the last couple years over at Canonical. Same thing, 5,000, 10,000 people and about six people in an office behind a 7-Eleven somewhere.Your background is amazing on both of you guys. Your current skills, your current projects and activities, what a pleasure it was having. I'm not throwing you out if you got more you want to add, by all means, but you gave us way more time than I had possibly hoped for, and I know.

Skip Grube:

Be careful. We'll talk your ear off if you let us.

David Rush:

Great. Everything at Rocky stops when you two guys get together and do something like this. It's catching the show and there's no development happening and there's no negotiations.

Skip Grube:

It's been quite a journey, honestly. When I got into this, I was like, I've done a little bit of packaging work. I was like I had not seen nothing. I would tell one quick story, one quick anecdote with my previous job. My boss and not Greg now, but this was before I worked at CIQ. It was shortly after Rocky Linux had got started. We were still bootstrapping the whole thing. We had nothing out there. I had mentioned to my job that I was working on this in my spare time for NDA reasons and that thing. They were like, sure what Linux Geek, whatever, do what you want. It should be noted too, that like in my previous job I wasn't the only Linux guy. But as far as infrastructure goes, I was pretty much the guy.

I know there's admins out there that have, have been there as well. I had just started working on this Rocky Linux thing for a few months, and it was with these amazing people over chat and seeing them nights and weekends and we're trying to bootstrap this thing. My boss comes around the corner and he's talking to I think my boss's boss or someone superiors. He was like, oh, and here's Skip. He knows everything there is to know about Linux. I was like, oh, honey no I don't. Like you don't know these people I work with the weekends, oh my God.

David Rush:

Skip, did you "Oh, Honey" your boss?

Skip Grube:

I did. In my mind, I didn't "Oh, Honey" out loud. In my mind I was like, oh baby, you have no idea. I said something to that effect to him. I was like, I think his name was Chris. I was like, Chris, you don't know man. I am nothing compared to this group I work with, and they're all just coordinated over the internet and they're all brilliant.

David Rush:

I'm looking at you and you are among the Giants on whose shoulders more will stand more do stand I standing on giant shoulders to boot. Ditto for Greg. What an industry giant, a technical, giant visionary is not an unfair term by any stretch.

Working as a Team [1:41:40]

Greg Kurtzer:

Skip is being really modest. While we love the modesty, he's also very accurate. I want to describe this a little bit. We had 10,000 people show up, like zero to 10,000, faster than I've ever seen anywhere in any open source project that has ever existed in my multiple decades of experience in open source. Never seen anything like that before. I was just lucky to be part of it. The amount of capability on the team I mean, you could imagine 10,000 people all volunteering to say, we can help. We split things into channels. We had teams like organically start developing. It became a meritocracy across the board, like a perfect, near perfect meritocracy, if not a perfect one in the sense that I didn't know Skip before, I had no idea anybody on the team.

Everybody was just like here we are, let's go to work. There was almost no way to organize it. There was almost no way to manage it. I mean, how do you manage it if you don't even have a management team. I cannot manage 10,000 people. I was completely overwhelmed and just we had the good fortune to, and I don't remember who said this, but let, it was, let's start breaking apart all of the different areas, the functional units of what it is that we're creating and what it is that we're doing. Within those, those were small enough focused groups that certain people were now able to people started working on things and a natural organic leadership started to form.

Again, I didn't know Skip. I think it was Skip in Slack, but I don't remember. I mean, even, I mean, so many people used aliases. Many people didn't post pictures, most people didn't post pictures, or they used cartoon characters or something else and whatnot. We had no idea who anybody was. All of a sudden we now have an organizational structure and we have a leadership structure starting to emerge, and we have things getting done. Now the organization at that point was very manageable. We had 10,000 people show up, but we can't have 10,000 people all contributing. That did filter down. The team that we got is so ridiculously amazing. The amount of intelligence capability and just humanity that is within the Rocky group is so tremendous.

I couldn't have even wished for a better group if I could have hand picked. It is fantastic. That is why from a CIQ perspective, I've tried to hire as many people in the teams that I can. Because at this point, I mean, everybody that's associated with Rocky is just so dang good. Just to put it bluntly. Just really good people. I've seen people come in from the community into our matter most or into IRC. Everything from trolls to newbies, to people that have never installed Linux, but want to get involved to even non-technical contributors. People that have never contributed to Linux or open source before, but they're like, Hey, I do this really good, is this of use?

Yes, it totally is. Let's figure out how you can join and how we can all work together. Again, it is all 100% community from the infrastructure going through the operating system, through the entire stack. It is all community, and the community owns it. The community controls it, and it has just been amazing to be part of this and to witness it. I really just wanted to give a big, huge kudos to every member of the team. We have some of the brightest and smartest technical people out there in the Linux distribution space, and we also have many of the best people that I've ever worked with there. Between that and CIQ, honestly, I'm so happy and so excited over everything that's happening. It's just fantastic. Anyway, I'm done raving and whatnot.

David Rush:

Well, I know they're looking at you saying you are not allowed to retire for at least the next 20 years now. You've got the energy, you've got the vision, you've brought them together, the confidence it's so clear from the timeline. You got four minutes and four months of infrastructure development, and then three months later you got a product release that's strong.

Skip Grube:

It was really, I was amazed myself, honestly, even part of it, I'm just like, oh, wow, we did all that. I was thinking about what Greg just said. I consider us amazingly lucky, but at the same time we had 10,000 shots. We weren't that lucky. 10,000 tries.

David Rush:

Well, for everybody else, one other fun thing for you guys besides help them out, okay, contribute, if you contribute a couple of bucks, if you can contribute some expertise. If you've got some skill, Rocky is a great place to use it. Check out their site, rockylinux.org. They got a store and I never paid attention to it until today. I got Christmas shopping to do before the weekends out. There's going to be some Rocky stuff. There we go, there's the cups and the backpacks and the whole ball of wax. Hit the store, help them out, guys, and go Rocky. I need beauty shots of Rocky doing cool things on RAS Pi. That's my shirt and that'll be my hat.

Greg Kurtzer:

Dave, I want to just send out a special thank you to you as well. I know Skip pretty well so I'm going to speak for him. We had a really good time. This was actually, this was a lot of fun. Please, if you ever want us on again, just say the word. I'd love to also bring in some of the other members of the Rocky team as well. Maybe at some point we just have a huge party.

David Rush:

Sounds good. I won't burden you too much, but count on it. Next year we're going to kill this show in two weeks. Drama, Dave Rush asked me if anything goes away because it existed to serve a niche that doesn't exist anymore. That was people studying for CompTIA certifications during lockdown. Lockdown is gone. We're back to normal procedures. But I've got a blessing from the company to start a new operation that's going to be PI and Linux focused. It's not going to be CompTIA study focused. You guys will fit better into the new module whatever we're going to call it. I think we're going to call it Andramaduh. We've got drama right now. Dave Rush asked me anything, so we're going to surround it and make it Andrama, and add duh somewhere. I don't know, something funky like that. As you experienced naming things has to be organic and it's got to be a lot of fun. But that's where I'm poking at.

Chat Rocky Linux [1:4933]

Greg Kurtzer:

Last thing real quick. Everybody watching this show, please, we have a lot of fun hanging out in MatterMost and being part of the community. If you're on IRC, just head over to the Rocky Linux channels. If you're not an IRC or you want something a little bit more slack like we use MatterMost. If you just go to chat.rocky linux.org, chat.rockylinux.org, you're going to find our chat system over there. It's running Mattermost. There's local clients, there's phone clients, and there's a web client as well. We'd love to have you join and just be part of what it is that we're doing. Help other people there be part of the community. We would love that. Please join, please be part of everything that we're working on.

David Rush:

Don't leave now, because what I have I completely ignore the chat. I get enraptured by what you were saying and so lots of good stuff here. Will Shaw is sold on Rocky is his next distro. I've been pushing him on that for a long time. Zerxisk is here, I need another machine to run it. What else? I saw a couple good ones roll by. I can't imagine building an OS though getting overwhelmed. Boy, you're exactly right. Zerxisk says, I've seen this on the Raspberry PI forms all the time. I'm a beginning coder. I bought a Raspberry PI. I want to write an operating system for it. Nobody writes an operating system anymore. You do what you guys do. You take pieces and improve them and fix them and support them and do whatever. But they're always gung-ho in the beginning until they realize what's really involved.

Greg Kurtzer:

By the way, Dave, while you're looking that up. You mentioned our merchandise store. We're going to send you, I'm going to have Jessica reach out. We're going to send you some shirts, some hats and whatnot. Please, please wear it on the show. if you want, give some away, we'll give you a bunch.

David Rush:

Absolutely do that. I always think about putting the bite on a guest and saying, Hey we got to give away contest here. What have you got? But I just didn't want to do that with you guys. It was a time issue and lots of others. Oh, Jason, you're two hours from Skip's house. You guys need to get together. Jason Helms is one of our strong regulars and one of the mods of the unofficial Total Seminars Discord channel. There's a whole bunch of the mods here. There's Asheville down the road. I didn't realize that Betlind, that you were from there. Andre is from the Netherlands, though he thinks for some reason that he is from Portugal.

He's not that close. Skip is responding in here. You went to high school in Asheville. Will is a Delta Airlines captain. He gets everywhere and visits everybody. I'm going to visit with him in two weeks when he gets into Houston next time. Tullowit is in fact on a small island in the Pacific. He's in Hawaii Paradise. I don't know how we go dragging down these things. There's always something that somebody can contribute. Little piece of conversation that gets everybody's trigger stripped. What do you got here? Skips talking to Linux Fest. While I'm looking in here, where's the next conference or two that we're going to see you guys at?

Greg Kurtzer:

Oh, that's a great question. I wish I can even just answer it. I don't know, , I don't know what's even coming up next. We have this huge list of all of the conferences that we're planning on going on in '23. I'm at a loss.

David Rush:

Get that website. That's something that I ran across here recently. Lots of people want to know where my boss is day-to-day. He's a big personality in the industry. Mike Meyers writes all of the top books in our Niche, and people call it, where are you going to be next weekend? Where are you going to be in? He's the same thing, I don't know, my administrative assistant knows that. They tell me the day they hand me my plane tickets. I'm within our organization to get a tab on the websites that says, well, here's where you'll find Mike. Maybe the same thing for the nice folks at Rocky. Here's where we're going to be.

Skip Grube:

Might be a good idea.

Greg Kurtzer:

Yeah, I'm with you on that, Skip.

David Rush:

I'm a contributor.. Yes, Alex would've loved this show. She's our photo section up in what's the Milan? She's up in Milan and a big Linux head and in Ninja.

Greg Kurtzer:

Dave Laduke mentioned that we're going to be at Scale 20x.

David Rush:

Okay. Scale 20x and join our Holy War. Pineapple on pizza or not?

Skip Grube:

Absolutely not! Easy question. No.

Greg Kurtzer:

Okay. I'm good with it. I've actually, sometimes I'm actually in the mood for it because it's just lighter pizza. Now granted, I'm not committed to it, like I can do any pizza. But here's another one that I usually ask because it tells me a lot about people. Pirates or ninjas?

David Rush:

Pirates.

Skip Grube:

Pirates. I'm too out of shape to be a ninja.

Greg Kurtzer:

Alright, we got the pirate crew here.

David Rush:

Pirates had their heyday, their movie heyday and their book heyday. Pretty consistent pirate characteristics were the same in the beginning to the end.

Greg Kurtzer:

I want to see how people vote Pirates or ninjas?

David Rush:

Get it out guys. Pirates or ninjas?

Greg Kurtzer:

We should have asked that at the beginning.

Skip Grube:

Show where they jumped from the hanging thing to the hanging thing. I'm like, pirates are for me.

David Rush:

We got bigger swords anyways.

Greg Kurtzer:

Oh, I get seasick too. I wouldn't actually want to be one.

David Rush:

One of our presenters and swag for all. I was thinking about that, yes, I am actually in a position to give away some Rocky Linux. You get a Rocky Linux and you get a Rocky Linux and you get it. Everybody should be playing with it in a VM or on a RAS Pi.

Greg Kurtzer:

Dave, we'll send you some swag as well. Feel free to send that out to whoever you like and, and enjoy.

David Rush:

We got more ninjas than Pirates, I'm sorry to say. Hey guys, thank you. I am honored that you were able to spend time with us at all, and so much thank you. Thank you to Jessica. Best to Rocky. I'm going to be a big cheerleader for you. To some level I will find a way to contribute too. It's been just one I had every bit as much fun as you guys did. You're lucky, you're fun to learn from. If I were leaving Total Seminar.

Skip Grube:

Thanks for having us to hang out. We like it.

Greg Kurtzer:

That was a lot of fun, Dave, thank you. Everyone who's watching and in the chat that was a lot of fun. I really appreciate how interactive it was and whatnot. Thank you.

David Rush:

All right. Merry Christmas. Happy holidays to you guys. Thank you, Skip. Thank you, Greg.

Greg Kurtzer:

Bye.

David Rush:

All right, bye-bye,. Now let's watch them be awkward for a minute while they figure out how to quit.

Greg Kurtzer:

This is always the time in which you're like, yes. This is always the time where it's like do we leave or do we just mute and wait and then go back into Green Room?

David Rush:

I'm just going to do what have I got about? Oh, I got two minutes of the end of the show. I got nothing to do, but say goodbye to everybody. I'm going to do that. Next week that it's the way I'm going to do this. I'm just going to do our usual shutdown stuff and talk about what we're doing next week and the following and do the grand departure here. Upcoming episodes next week, I have two developers coming in from Mixxx, the FOSS DJ software runs on Linux boxes, runs on Windows boxes. We did an episode on how to install and use Mixxx. Now we're going to talk to a couple of the developers. The lead developer's not available. Mr. MIT super brilliant also has so many polls on his time strings that he sent me a couple of his top developers, so we're going to talk to them.
Then we got Christmas and I don't know Mike is going to do a regular show this Monday and on Wednesday. Then we've got Steve coming back the Wednesday after that on the 28th. Then we're going to close down Drama on the 30th. But again, we got a replacement coming up. First of all, my incredible gratitude to Greg Kurtzer and Skip Grube from Rocky Linux, they were wonderful. We will have them on again next year on Andromeda or whatever the new name is going to be. My thanks of course to Mike Myers, my former boss and now coworker and everybody at Total and National Cyber Group who so generously provide us with time and resources for us to get together here every week. My most heartfelt thanks goes to you and everyone who comes here to participate. I am Dave Rush, the senior instructor at Total Seminars and resident Pi Specialists. Wishing you a great weekend, take care of each other, take serious steps to stay healthy, and if at all possible, call or visit your parents. Never forget, technology is great, but the greatest resource we have are you and I. With that good night. I will stop in on the Discord channel in a little while and I'll see you on the AMAs next week. Until then, bye guys. Stay online and I want to hit the button here. I am out of here, later guys.