Featuring
Sponsors
FireHydrant – The alerting and on-call tool designed for humans, not systems. Signals puts teams at the center, giving you ultimate control over rules, policies, and schedules. No need to configure your services or do wonky work-arounds. Signals filters out the noise, alerting you only on what matters. Manage coverage requests and on-call notifications effortlessly within Slack. But here’s the game-changer…Signals natively integrates with FireHydrant’s full incident management suite, so as soon as you’re alerted you can seamlessly kickoff and manage your entire incident inside a single platform. Learn more or switch today at firehydrant.com/signals
CrabNebula Cloud – CrabNebula Cloud is here! Distribute Tauri apps and Electron apps with best in class updater. At the heart of CrabNebula Cloud is a purpose-built CDN ready for global scale, and secure updates as a first-class citizen. Learn more at crabnebula.dev/cloud
Tailscale – Adam loves Tailscale! Tailscale is programmable networking software that’s private and secure by default. It’s the easiest way to connect devices and services to each other, wherever they are. Secure, remote access to production, databases, servers, kubernetes, and more. Try Tailscale for free for up to 100 devices and 3 users at changelog.com/tailscale, no credit card required.
Notes & Links
- The Oban Pro featuring Parker Selbert (Backstage #23)
- Robust Job Processing· Oban Pro
- Elixir and Phoenix with Chris McCord (Changelog Interviews #147)
- github.com/thechangelog/changelog.com
- Hatchet
- Have an Enemy | Getting Real
- Mojito island is a mirage – Signal v. Noise
- Attack of the Canaries! with Haroon Meer from Thinkst (Changelog Interviews #557)
- Cronitor
- Buzzsprout
Chapters
Chapter Number | Chapter Start Time | Chapter Title | Chapter Duration |
1 | 00:00 | Let's talk! | 00:38 |
2 | 00:38 | Sponsor: FireHydrant | 02:30 |
3 | 03:11 | Bogo & Friends 👀 | 01:13 |
4 | 04:24 | The Oban Pro | 01:29 |
5 | 05:53 | On freedom numbers | 00:57 |
6 | 06:50 | Explaining what we do is hard | 02:20 |
7 | 09:10 | "It's just background jobs" | 01:45 |
8 | 10:55 | When incidents become "incidents" | 02:57 |
9 | 13:52 | How Sorentwo got here | 03:49 |
10 | 17:41 | Oban in Changelog code | 07:27 |
11 | 25:08 | Sponsor: CrabNebula Cloud | 02:48 |
12 | 28:15 | Elixir as mature niche | 03:32 |
13 | 31:47 | A fork in the road | 03:12 |
14 | 34:59 | The Hatchet path | 03:35 |
15 | 38:34 | Naming servers on an island | 01:04 |
16 | 39:38 | So what's the answer? | 04:38 |
17 | 44:15 | Similar situations | 02:32 |
18 | 46:47 | Almost an SV reference | 02:16 |
19 | 49:03 | The business model | 01:14 |
20 | 50:17 | Sponsor: Tailscale | 02:53 |
21 | 53:21 | Reaching the freedom number | 03:40 |
22 | 57:01 | How we use Oban | 03:26 |
23 | 1:00:26 | Jerod's bespoke Oban feature | 01:08 |
24 | 1:01:34 | Parker's awesome PR | 03:08 |
25 | 1:04:42 | Cron job feelings? | 01:36 |
26 | 1:06:19 | Shout out to Cronitor | 01:01 |
27 | 1:07:20 | Shout out to Buzzsprout | 01:11 |
28 | 1:08:31 | Indie / small shops | 03:27 |
29 | 1:11:59 | The "Lifestyle Business" | 06:22 |
30 | 1:15:22 | A theory on software biz growth | 02:58 |
31 | 1:18:20 | No more worlds to conquer | 02:38 |
32 | 1:20:59 | Alexander the Crybaby | 00:36 |
33 | 1:21:34 | Adam's essential Calm Balm | 00:48 |
34 | 1:22:22 | Business mantras | 02:01 |
35 | 1:24:23 | Gaining perspective | 01:14 |
36 | 1:25:38 | The house that wasn't | 03:22 |
37 | 1:29:00 | Big change brings big change | 02:35 |
38 | 1:31:34 | Retirement talk | 02:25 |
39 | 1:33:59 | Bye friends | 00:44 |
40 | 1:34:43 | Coming up next (join ++!) | 01:24 |
Transcript
Play the audio to listen along while you enjoy the transcript. 🎧
So we’re here with Parker and Shannon. This is our BOGO episode, buy one, get one. So we have Parker here with us, but this is not just Sorenone, this is Sorentwo. We’ve got both Parker and Shannon. They are teammates in multiple respects. In how many ways are you two teammates?
Gosh… Start counting.
Let me count the ways.
Business partners…
One.
Spouses…
Two.
We made children together…
Okay, three.
Multiple times. I don’t know if that’s more than one…
Keep it clean…
Yeah. So I guess that’s two and a half…
She’s anticipating four. You’ve gotta have four.
She has her fourth finger up.
She can pitch in a fourth, if you have an idea.
Yeah. Hey, teamwork on this question. That’s a fourth.
Podcasters together…
He does insist upon that… He does. A friend. I would say he’s a very good friend to have.
That’s true.
There’s my silver lining.
Best friends as well. So that’s a team of sorts.
He depends on me heavily for comedic effect.
Yeah. Parker sets them up and Shannon knocks them down. That’s what we’re gonna expect from here on out. That’s a high bar, Shannon.
That’s a big – that’s an order that I cannot maintain.
I think you can do it. You’ll be fine. Well, this is Changelog & Friends. We’re hanging out. Parker we’ve known for a while, because you are the creator, maintainer, purveyor of Oban, the background job processing library for Elixir, which we are users of. And you have Oban Pro, which you’ve been having worked on for a while now. I had you on I think Backstage - it was just you and I, don’t think Adam was there - a couple of years ago, talking about Oban Pro and your desire to go full-time Oban, or retire via Oban, I can’t remember what it was… But there was a freedom number involved… I think you still had a – is that because you still had a day job? Help me – jog my memory here about your status back when we had you on the show.
Yeah, I mean, your memory is correct about all those things. I think it was probably a year and a half ago; it was just you and I. We were talking about Oban, and Pro. We did talk about some freedom numbers. The ultimate goal, we call it our retirement project. I still do have a day job, with Dscout…
Oh, you do.
…for a variety of reasons. One, just kind of – I’ve been there for a long time and I have a responsibility to the founder, who’s a good friend and longtime co-worker. And then also, it’s such a huge user of Oban and Pro that pushes it, that I just have daily insight into exactly where people feel pain points, and what people need. So yeah, it’s still going.
It makes sense. So what is the retirement situation? What is the freedom number? What is this?
I think it’s a ruse…
Okay. Tell more.
How so?
Because I don’t think he would ever want to be free, really.
I think the definition of freedom is important here.
What is freedom? Right. So passion, the project… Oban is in every aspect of our life. I mean, it’s in every dinner, it’s drawings on the shower… It’s in every aspect of our life; we work on it. And I don’t think I realize that until we leave our little bubble, and we’re around other people. And we get this mirror image of what we’re doing all the time, and how often we go back to it. I don’t think I’m aware of it until we’re around other people outside of our bubble.
So you’re with your friends and they’re like “Why are you talking about Oban?”
I thought it would be hilarious… You know, when you do this “What if” in your mind, I thought it would be hilarious for most developers to create this video of our kids trying to explain to our neighbors, or to anybody else, what it is that we do, and get them a whiteboard, and have them try to draw it, or have them try to describe what it is that we do… Just somebody else who sees us all the time, communicating what it is they think we’re doing all the time.
It doesn’t have to be kids, it can just be me trying to explain to people what we do… Because I do a terrible job.
I think you do a great job.
I just bail out.
He gets so deep, so quickly…
I take the opposite stance; I try to stay as high-level as possible. So my old line was “I make websites.” And that worked pretty well. People were like “Okay.” Because they understand websites. “You make websites. We’re done here.” And then I got to writing other stuff, and so I changed it to like “I write software”, or “I’m in software”, and that was pretty much people are done with you at that point. They’re like “Okay. That’s nice, Jerod. Let’s move on.” Now it’s weird, actually. “I podcast about software.” And they’re like “You what now? Because those are two things that are foreign to me.” And it usually opens up a bigger question. They don’t want to ask about the software, but they do want to ask about the podcasting. For some reason that’s interesting to people… But also sometimes harder to explain how it all works.
[00:08:13.22] It is.
You’ve got a meta situation too, because you write software to make it easier for you to publish your podcast about writing software.
Yeah, exactly. Which - I never go there. Like, I can go there with you guys.
It’s a Russian doll. It’s just a Russian doll situation.
For sure. And even telling people what Changelog means, that do not understand software. They’re like “That’s kind of a weird name.” But in our world, that’s kind of a perfect name. But they’re like “What is the Changelog? What is Changelog? What is that?” And I’m like “Oh, well, that’s –” And I have to explain what a changelog is. And it’s like “Well, this is the difference between one version and another. It’s how people communicate what those changes.”
And they’re like “Your podcast is a changelog?”
And I’m like “Well, if you were in the world, you would understand how on point the name is, and how perfect it is… Unfortunately, in this moment, you’re not, and so therefore…”
[laughs] “You’d be really impressed with our name if you were in this world…”
Well, then it’s like you’re explaining a joke. And you can never explain a joke, because the joke’s ruined.
For sure.
Once you start explaining it, it’s over.
It goes wrong. Really bad. So Oban is discussed a lot at your dinner table… Or just, I guess, generally, everywhere where you’re at.
Right.
How so? I don’t want to say this pejoratively, or as a pejorative, but like, it’s just background jobs. Like, how much can it be in your conversations? [laughter]
It’s just a dev tool.
Yeah. Sorry about that. I had to do it.
No, no. If you phrase it like that, I think it makes perfect sense. But it’s not just our background jobs. It’s a lot of other people’s background jobs.
Okay…
There’s an open source version, and then there’s a paid version, and the paid version comes with support… Although, quite honestly, we’re not great about just leaving people who don’t pay for support to the wolves… It means that there’s just a lot of helping other people with their problems with backgrounds jobs. And invariably, when you build a tool, which can be pretty complex, you can’t predict all the ways that people are going to use things, or the environments they’re going to use the things in, and you can’t test in all those environments… And you put all those together - it can be pretty deep for debugging, or just support. So you kind of get to know people. Some people come back more, they just have bigger issues, or they’re just a little chattier about the issues… And so it’s not always that we’re talking about Oban as much as the people around it.
It’s exciting when you have a new customer, that you’ve used their product in the physical world, and you see their name come up, that suddenly they’re now a customer… That will spark a conversation. Being responsive to our customers, and having them give feedback and accolades will lead us to another conversation… It takes very little. We take the bait.
Yeah. I think what’s really important here is the partnership in the opportunity, right? Because - I don’t know how you are, Jerod, with your work and your wife, and I know how it is with my wife… She’s not deeply involved in our day to day, and so we’re less partner in the things we do day today… However, she’s obviously deeply invested, because hey, we’re husband and wife. And the same for you and your wife. But you all are working directly together on the output, on the effects, you’re sharing in the ups and the downs… Which makes the journey, regardless if it’s up or down, just a bit more fun, and a bit more supportive, I suppose… You don’t feel like you’re falling solo; you’re falling together. Or rising together. Which to me is kind of a beautiful thing.
It stands out where an incident could very easily become an incident though, if you’re not careful… And that same – I agree with you, and the positive is there.
I thought it was poetic. I liked that.
[00:11:49.21] It is poetic, and I embrace that. But I hate to give somebody the whole pie in the sky without the reality of the patience it takes to restrain yourself when your partner is refactoring something, or has critique of you in a response. And the familiarity leads you to a dark place. You have to be patient; you just have to be extremely patient… Because a work incident, an incident with a customer or with code - it could become a personal incident in a matter of seconds. So it’s just that restraint.
Yeah. Well, that’s where love and respect comes into play, right?
Absolutely.
As partners, I think, even from my wife, and I’m sure, Jerod, you’re the same, because we’re kindred spirits in a lot of ways, it has to begin with love and respect, right? And if you’re formed there with goodwill, like I have goodwill for you in the formation of the partnership at the friend level, then the husband and wife level, and then the kids level, and then the business level - like, if all of those things are founded on love and respect, I think even those incidents are a little easier to deal with, because you do the 10 second rule. “Hey, I’m upset right now. I’m going to pause, I’m going to count the ten. I might even go away and count to 100 if I have to, if it’s a really a big incident…” But love and respect sort of gives certain contracts, I suppose, into place, because of the foundation of love and respect. And I’m assuming that. That’s my foundation with my wife, so I’m sort I’m just sort of assuming that’s for you all, and giving it to you if you don’t have it.
No, I think we do have it.
I think it’s a good assumption. Yes, we spend a tremendous amount of time together. Possibly a disturbing amount, to some couples. We were doing it before COVID, before people so frequently worked from home… But not only are we working in the same space and living in the same space, and then raising kids in the same space, but working on the same things, in the same space. And you have to have a tremendous amount of respect.
Yeah.
It makes sense to me how you got there on the other dimensions, but on the work dimension, on the business, how did you end up here? Were you already business partners, and this was the opportunity? Did Shannon start and Parker hopped on, vice versa? How did it end up to be this partnership around this project?
I had been consulting originally, and started my own business, and wanted Parker to consult with me. He had been working at a design firm. And with the birth of our second child, we decided we were going to take the leap and do it together. We had tried to explore some other app ideas, and had made it to a very shallow level, especially with something like Snow, and things like that. And it just didn’t really take off. We wanted a passion project. We both started with Ruby on Rails, and Parker really made the brilliant choice to move to Elixir. And I think he has led us there. So I may have started it, but he’s led us to where we are, in a wonderful way.
And you had a proven guinea pig, so to speak, with Mike Perham and Sidekiq, and Sidekiq Pro, in the Ruby world, in the Rails world. Much success, very much similar projects, similar structure. That had to feel good, to say “Okay, here I am in a different place, with different tools, probably some of the same people who remember Sidekiq, like me, for instance… So there’s an analog that I can draw pretty easily to Sidekiq.” Although in the Elixir land background jobs are less necessary. The kind of background jobs that you provide are less necessary for people just getting started. And then I think eventually you kind of do grow into wanting the persistence, and a lot of the scheduling and the tooling that you provide… But I went a long time without having any background solution on our application. Probably three or four years. I need to look back and see exactly when I pulled Oban in the first time. But besides that, you did have kind of Rails to run on, to a certain extent. Did that give you confidence? Were you basically saying “I’m going to Mike Perham’s playbook and apply it in this ecosystem”? I mean, no shame in doing that, by the way. That’s a good way to do things. Was that part of your playbook?
[00:16:16.15] That was definitely part of the playbook. And I do have history. I’m not gonna say Mike’s my friend, but I do have history in the past of collaborating with him, and sharing some work with him, and writing a tremendous amount about Redis and Sidekiq, and helping former contracting clients onboard Enterprise and Pro. So I was deeply familiar with both the business, and the code, and all the features, and saw it as such a great way to have a technical project that is a – it’s a developer tool, so you’re selling to developers, but also scratching your own itch, where you understand deeply exactly what problems you have.
But you’re right, in many ways it’s not as necessary in Elixir. You can get away without it for a while. And that until you understand that you have certain guarantees that you want to hit, of “I’m not ever going to lose this thing.” So the concurrency story in Elixir compared to the concurrency story in Ruby are vastly different. It’s unbelievably easy to do certain things in Elixir, that would be tremendously difficult, if not impossible to do in Ruby. And so that’s part of like the springboard for – Oban has a lot, I think, deeper features than much of what even Sidekiq Enterprise still has. And that’s purely because of what Elixir makes possible.
But I will say one other thing is that I remember after somebody added Oban to the Changelog codebase, before that you guys were using Quantum for Cron. And so in a way, you were using a Sidekiq Pro kind of feature, but with a different project there.
Yeah. We needed scheduled jobs, and I reached for Quantum for that. But in terms of running emails and stuff like that… Like, the kind of stuff that you would immediately in Ruby on Rails land you would reach for Sidekiq, pretty much immediately, as soon as you want to send an email in the background without blocking. With Elixir, even our stats processing, a lot of the stuff that we do in the background using Oban today - which we can look at all the things - I was just doing that with straight up Elixir, and the beam, and all those fancy tools… And it’s not fancy; it was easy for me to do that. But once we wanted scheduling, and then retries… Even just visibility into – once your background jobs really matter, then it was like “Yeah, we want something that’s going to provide more than the built-in.” But like I said, we lasted probably three or four years without it, and we’ve probably been on it for three or four years, something like that.
Yeah. And I think when you were starting, Oban didn’t exist. So you couldn’t have even used that if you wanted. It would have possibly been something else.
Yeah, there was a lot of stuff that didn’t exist… And so I hand-rolled a lot of solutions, which are still in our codebase today. We get questions sometimes, like “Why did you do that this way? This seems really convoluted.” I’m like “Because that was just how I figured it out in 2016.” There just wasn’t – like, Phoenix didn’t provide a way of doing that, or there was no best practice style library that everybody uses. I’m thinking of CSRF stuff, and a lot of the things around Turbolinks, and just cookies, and junk like that, where you’re just doing it yourself, you’re just handrolling it. Auth… I think I handrolled some Auth stuff, because it was just the early days.
So don’t necessarily go look at the Changelog.com codebase for how to do things today… But even context. Like, Phoenix really promotes the idea of contexts now, and I just ignore them all together. And so I tried to introduce contexts as the feature came out, because the Phoenix team was really big on it, and the ecosystem was big on it… And for our application it just felt like another layer of abstraction that I didn’t necessarily need, and so I just ignored it. Which - I love that about Elixir; you can just ignore the stuff. And about Phoenix, the way it’s built. And you just don’t have to use it. I mean, sure, it’s the idiom; you’re gonna look less like a Phoenix app by doing so. But you can just ignore it and do things your way. And so I’ve continued to do that, and I think that most modern Phoenix apps wouldn’t look like ours, at least for that reason. And LiveView.
[00:20:28.07] Yeah, yeah. Plus LiveView. Yeah, for Oban web, for the development, we have like a single-file script that’s essentially an entire Phoenix application in one file, that also generates random jobs, and all this different timing… The kind of stuff that we have on our demo, so that when you’re working on it locally, you have some reasonable amount of job traffic, and errors, and cancellation. But that fits in exactly one file. It’s not like it’s an entire, full-blown, scaffolded Phoenix app. It’s just the bare bones things. And as long as you configure it right, and the modules are there, it all just works. So it’s easy in that way, once you understand what the parts are.
Right. Well, that was one of the things that attracted me to Phoenix in the first place, was how simple it was relative to Rails, in terms of – for me, it was the stack traces when I threw an error on the page. The stack traces were tiny. You know how in Rails - probably to this day; I haven’t done Rails in a long time. But it’s like “Here’s your application code trace, and then here’s the framework trace. And this one’s collapsed by default, because you don’t want to look at that one.” And it’s the same way in Phoenix. But when you go and expand the framework trace, at least back in the early days, it was like eight function calls. I mean, it was nothing. My code was there, and you could see the entire pipeline of function calls. And it was like 12 function calls… And I was like “Wow, this is something I can actually reason about relatively easily.” Now it’s probably a little more deeper than that now. You guys probably know better than I do, perhaps… But I liked that. It was just, it was grokable. Like, you could see all of Phoenix, and there was nothing hidden from you.
I know it’s a little unorthodox, but did Phoenix bring you to Elixir then? Because that would be –
Pretty much. I mean, Chris McCord basically brought me to Elixir. So in that way, yes. We had Chris on the show before we had Jose on the show. We were just reminiscing with Jose a few weeks back about this… Because I’ve been a web developer pretty much my whole career, and Rails for many years… And Chris had a very similar story to mine, but then he found Elixir, and he built Phoenix, and he came on the show, and he talked about how cool it all was… And I went and tried it, tried Phoenix specifically, and got this little application up and running in like a couple hours, and deployed as well… Which was just a simple one endpoint, call Slack API, do a thing. It was like our old Slack inviter before we had a full system. And that was just so fun and easy, and it looked somewhat like Ruby… And it turns out it’s nothing like Ruby, but it has this facade of Ruby-esque taste; maybe it has a Ruby taste to it.
It’s got a slight syntax to it.
Yeah, exactly. There’s a flavor there.
A veneer.
And I was like “Oh, this is pretty cool.” So it just proved itself relatively quickly. And back then, I was a full-time contractor, part time Changelog. Or even just maybe nights and weekends Changelog back then. Not nights and weekends technically, but just spiritually… Because I was an independent business owner, so I could work on it when I wanted to, so it’d be during the day times. But it was a hobby for me. I think Adam had gone full-time by then. 2015 was your full-time?
I’m looking at the date when we recorded that. That was – March 20th 2015 was the record, and shipped it, which means it’s probably around the same week span… Because this is prior to us having separate date fields in our CMS for those things. Because this was published probably the old way, prior to the CMS. And the CMS is the application we’re talking about.
[00:23:58.02] Oh, it had to be the old way.
Yeah.
On stone tablets, is that the old way?
You were hammering it out?
Basically, yeah. Well, I don’t think I went – I think I went full-time around that time. I think it was like February 2015. So this was like literally when we were courageous enough to say “Adam, quit your day job and make this thing your thing.” And so I guess we’re coming up I guess next year would be 10 years of being full-time on this thing.
Wow. That’s a great anniversary. We celebrated our five-year recently…
Congratulations.
…which was big for us.
There you go.
Not as a people, but for Oban.
In which context? Come on. You have to set it.
Dual anniversaries here.
Yeah, dual anniversaries for the Oban context. I think that would be a dream, to have people choose Elixir in the way that people love Phoenix. I’m not going to say that would ever happen, but that would be a dream, that people would come to Elixir, wanting to use Oban.
Break: [00:24:56.23]
We understand our niche pretty well. We can clarify the market all we want for us, unless we have more people, more Elixir users, we’ll max out at some point. That’s just the reality of such a – Elixir is just an amazing thing…
But it’s a niche.
But it’s a niche. So we have to – we’ve discussed recently doing something to just try to widen that pool and bring more people to it.
Yeah, I think that’s interesting, because I think it’s not only a niche. It’s a successful niche, first of all, but it’s also a mature niche at this point. I mean, it’s a pretty mature language, and a mature ecosystem. And I think there can be new tools that bring people on. I think Nerves probably brings people in. I think LiveView itself and a lot of the stuff that Phoenix is doing is cutting-edge… And now you have - what’s Chris doing at Fly, with the serverless stuff?
Ah, Flame/
Flame… He’s still pushing on the edges, which does bring interested people in. But that being said, I think Elixir, in a certain sense, and you guys can disagree - Adam, maybe you have a different viewpoint - has kind of found an audience… And so I think it’s kind of like at the top end of the S curve of adoption, perhaps. Unless ML changes it. I mean, it has great ML tooling. Jose has been investing in that. But most people are using Python, and newer things. Rust and Go… And so I’m wondering if the Elixir ecosystem has a lot of room to grow, or just some room to grow at this point? Because it’s pretty mature. What do you guys think?
I think – to rewind slightly, there’s the whole notion of like a killer app. The thing that brings somebody - like “I wanted VisiCalc, so I bought a Mac, or an Apple”, or whatever. And I think for a long time, Phoenix has been the killer app of Elixir. And we’ve had other ones, like you’d mentioned with Nerves, a little bit. And then I think the ML is opening a whole new realm of possibilities to attract people, that would have been firmly in the Python camp. And we sort of dream that Oban is also another companion killer app. There’s enough compelling stuff here, enough compelling features that it’s worth it to pick up and make an Elixir app just so you can use Oban, and the workflow stuff, and Pro, or all that. But even more so, if you were to pair that with the ML side.
So if you look at some popular tools in the Python side, like Dagster, or things that are meant to run workflows, they use them for ETL pipelines, or complex machine learning flows… And all those things do decently in Python, because people have put so much time into it… But it’s not as natural a fit as it is in Elixir.
So the vision that we have is that it’s enough that you can have your web app, but you can also have your web app orchestrate things, and it can do all this ML stuff, and it’s all under one blanket of a single ecosystem. And then you have all of the other, like fantastic documentation, and great packages, and a really vibrant, responsive community. You have all of those things, instead of having to scatter your attention between the JavaScript for some stuff, and a server-side language for some stuff, and Python for the ML side.
[00:31:46.00] As you describe this, I’m thinking, do you produce any content around this attracting folks to Elixir because of this comparative nature to Python, and how it’s good, and it does it, but it doesn’t do it well? …I’m paraphrasing what you said there… But it sounds like you’ve got a bias, obviously, because we know where you live in terms of where your camp is at. I’m just curious, how much do you help with the growth, not just provide the tooling? And I don’t mean that in a negative just, but how much are you doing activism, and outreach, and advocacy around the depth of ability that you can have in Elixir over other traditional ways, or maybe ways that are being seen as… Well, Python’s the easy path in some cases for those kinds of workflows, for example. And that seems to be the default in the marketplace, so to speak, of dev land. And they don’t consider Elixir, because there’s not enough awareness.
Well, should we discuss that which shall not be named? I mean…
What is it? [laughter]
You surprised all of us…
Alright, deep breath… So, I mean, I think you’ll see more content… So to your point, maybe not as much as we should have up until right now. We have not. I think you will see more content from us supporting that and doing that. We had researched a bit and had decided we were going to embark on a cross-platform kind of version of Oban. We were putting our toe in for Python, and I kid you not… Go ahead…
Oh, yeah. It was within, you know, a matter of time before Hatchet was – things were announced. There’s been a lot of movement in the past year.
It was just a Y Combinator effort was just launched. And their marketing is phenomenal. There’s a lot of lessons to be learned from that… But it was within weeks of us starting to strategize and pull together on that. And we had had a lot of walks and talks about how we can really do what you said, Adam, to really bring in our efforts towards the Elixir community, to support that, to… What niche can we fill within that community? What can we develop? What’s missing? What have we wanted? We’ve jotted down ideas… But we had started exploring a cross-platform version of Oban, and within weeks it was released. So I guess it’s just whether or not we’re going to put our efforts into picking a fight…
I’ll say – maybe not picking a fight. But the number one takeaway was - well, I guess there are two takeaways.
Competition.
Right.
It’s that the description of what they’ve built - it sounds fantastic, but the complexity of what they’ve built… You couldn’t even tout those things in Elixir. They’d be so table stakes to do things. But the marketing that they have around it, and the use cases they’re describing are things that would work perfectly for what we want to build. And definitely, I think there’s – we have to do work to try to communicate and bring people in.
I’m looking at a Hatchet now, and I kind of want to preface some things, because it seems like a startup, right? Backed by Y Combinator.
Totally a startup, yes.
Is Hatchet a cross-platform then? Is it focused on – it’s hard to tell from their description, because it seems overarching, not “Hey, we’re in Python land. Come get your Python task queues here”, etc. It seems very much like a larger play than simply one language or one camp.
I think it is. I think they’re doing sort of a perm factory kind of thing, where there’s a server that coordinates thing, and then they have SDK is in various languages.
What’s stopping you all from doing this? I mean, this – again, I think Jerod asked before, “Hey, did that give you confidence because of the path of Sidekiq and Sidekiq Pro when you initially started?” This to me says there’s room for a mature option, I suppose. But then that really depends on – because your whole entire operation goes from… Let’s just call you mom and pop. Do you mind if I call you mom and pop?
[00:36:07.19] No, we definitely refer to ourselves as the mom and pop software shop.
Right. I mean, you very much are. I mean, I don’t know more of your employment, like if you have other folks working with you… Maybe you have temps, I don’t know… Who knows? I don’t know what you have. But it seems mom and pop. It would say “Okay, you’ve got to graduate to a whole new level. And do you want to be at that level?” You may see the market opportunity; that’s going to change your lives, that’s going to change a lot for you… And sometimes - I’ve even lamented about this with podcasting, running an indie media company. It’s like, I like what we do, I like what we do as an indie media company… But at some point the market and maybe other competition may say that we, Changelog Media, has to become more of a “behemoth” of sorts, or funded, or operate differently because competitors compete with us for ear share, market share, mindshare etc And then so we’ve got to follow suit, or wither on the vine and die, or just be happy with our little slice of pie. Not intended to rhyme that, but I did.
So Y Combinator-backed, Hatchet seems pretty – just going to their page, it seems like it’s pretty strong as a competitor, or at least has the firepower of Y Combinator. You never know, really. I mean, this could be a beautiful paint job, who knows. What are your thoughts on what it might do for your evolution?
It is a beautiful paint job. I think there are a lot of lessons to learn. I mean, that was a moment, right? That was a moment last week, to see that, discuss it…
And I think – I mean, you mentioned being an indie media provider… I don’t remember the exact phrasing, but I think that’s –
Indie media company.
Indie media company. But that’s kind of the beauty of what your company is. And I think part of – our size, and how deeply embedded in the Elixir community we are, is one of the strongest assets for our company. And I think if we were just on the outskirts, and tried to push in to say Go or something else, without that deep history - I don’t know what it would take to get there, or even if we would have the attention or time to do it. We don’t have any temps. We don’t have any other employees. There’s nobody else doing support when we’re on vacation, or asleep, or something… You know who you’re getting.
We’ve never contracted out for Pro, ever.
Or web.
Or web. So, I mean, I could say hi to David… Hi, David. I know you’re listening.
For the server.
For the server.
David is the name of your server?
No, David [unintelligible 00:38:50.24] I’m saying hi to David…
So Lismore is the name of our server, and Lismore is the island that is across the bay of Oban. So there’s a fun play there.
And that’s where we keep the people that contract. They stay within that island. They don’t –
Nice.
Is that like Alcatraz, or…?
“Stay there, on your island!”
I mean, you could swim…
It reminds me of Son of Anton… Gotta do the ding, because that’s a Silicon Valley joke right there. Son of Anton was the server stack that Guilfoyle stood up whenever they got blacklisted from Cloud, essentially. Gavin Belson was like “You can’t play here anymore.” They got blacklisted from [unintelligible 00:39:26.05] Rackspace. That’s how relevant this was back then, in those days of this episode… But Son of Anton - I love naming servers, but that was the name of the one in Silicon Valley.
That’s brilliant. I think all the questions you just asked were the questions we were asking each other in the last few weeks.
Well, what’s the answer?
We have different answers.
Do we?
Let’s hear them all. Let’s hear all the answers.
Yeah. As a rap. And go! [laughter]
It must rhyme.
I can see the dynamic already playing out. You asked her to rap her response, and “Go!’ That was cool.
[00:40:02.01] And I took my three fingers and put them in the middle of my forehead, to just try to express the pain…
Alright, the answer… I don’t know. Well, you can say whatever –
Our answer today was similar to the answer four days ago. However, the answer four days before that was somewhat different.
[unintelligible 00:40:22.10]
I just want to know which one you want me to go with, because I –
Should I do a Tom Cruise thing here? I mean “I want the truth!” “You can’t handle the truth!” Give me the truth. What’s the truth?
As of this morning, our roles - we’re pretty defined in our roles. And hiring more people in a way that changes our life… Right now, we are happy with what we do, how we’re doing it… We feel a huge amount of responsibility to our customers, and to that trust that they placed in us. That doesn’t mean that they wouldn’t make other choices… I’m not foolish enough to believe that. So I think we have to really pivot and decide if we’re going to put our passion into building up that community, or if we are possibly going to try to compete and pick a fight. And I only use pick a fight as a term from a book. Not that I mean that.
In the old 37Signals “pick a fight” [unintelligible 00:41:34.22]
In a 37Signals kind of sense.
In a Rails vs. Java kind of way.
Good reference.
So that’s what I mean by that. I don’t mean a literal fight. So I don’t think we have a definitive yet. We very much like our roles, we’re very good with our time management, but all of these things don’t point to which direction we really are going to go yet.
Yeah, one thing it says actually in that book - I’m pulling it up - Chapter 8: Have an Enemy. And I remember actually reading this; this brings back memories.
It’s a great book.
Because I think a lot of what Basecamp – they were like “We don’t know what we want to be, but we know what we don’t want to be.” And so the enemy was very much like Microsoft-based tooling, and how terrible it was.
This is how I feel about this right now. I’m channeling a bit of this book right now.
Get it. Feel it.
Yeah. I feel we know our roles, we feel responsible to our customers. We might pick a fight. However, we know who our real enemies are, and our real enemy is not necessarily at this point a startup… That could be wrong…
I’ve written lots of JavaScript in anger, I’ve written a tiny bit of Go in anger, and I’ve written plenty of Ruby and Python. I’ve thread pools, and process pools, and done all that… None of that is pleasant. I don’t ever want to do any of those things again, unless I really have to. And by comparison, working in a BEAM language and working in Elixir is just so simple and joyous… That the idea of leaving that to go focus on something else is – it’s kind of painful.
So I think you know his answer. I think you get – his answer is very much “No, I do not want this.”
I’m all full of feelings and maybe not so much logic.
That’s alright. They’re important feelings.
That’s a good awareness to have in a moment like this. When you’re in that “We must pivot or consider pivoting” moment, it’s important to have awareness of truly the emotional response you have. And then counting to 10, or counting to 100, or maybe even 10 days kind of situation. That moment of emotion to decision, you’ve got to put some buffer in there as just a way to protect yourself from being too irrational, potentially irrational.
Right.
[00:43:58.27] Yeah. But also, for you as an indie media company, there’s part of it, like, how much do you want to grow, versus how much do you not want to face an existential crisis?
I think Jerod and I can answer that pretty well.
Okay.
I mean, I don’t know how Jerod would answer it… But I think we’re happy doing what we do. And I don’t I think we’re on the long game. Like, when I talk to people, they’re like “Wow, you’re really thinking–” Like, I was just at a “mixer” for GitHub. It’s here in Austin; they were here for South by Southwest, and they invited me out to come hang out there for a couple hours while they were at the garage bar, and I was there… And I was talking to folks, and I was like “Hey, we should pod about that sometime.” They’re like “Well, I work at Apple and I can’t tell that story.” And I’m like “Well, we’ll be here whenever you’re ready. That sounds pretty cool, whenever you can share.” And they’re like “Wow, you have like this long game approach.” I’m like “Well, I’ve just been doing this for so long, and I have a pretty good confidence in like how we’ve been here, and what keeps us here”, both individually as passion, but also as a company… And I just have confidence that we’ll be here, at least in a few years, to maybe have that conversation.
So they were just surprised that I had this long game approach to things. And I think you kind of have to. And so our long game approach isn’t “OMG, there’s a competitor, or there’s a podcast that’s more popular than US, or there’s an influencer who’s more influential…” Jerod and I are not trying to be influencers or influential. We’re just literally trying to talk to people who share a passion like we do, get excited about where software’s going, and just have amazing conversations. And from a technical level, produce a really well-produced podcast it sounds like people want to listen to. We produced a beats album with Breakmaster Cylinder, we have music out there… What other job could Jerod and I do this kind of stuff at, on our own whim? We make the choices. We get to go into the nooks and crannies and hang out there, or zoom out and get bigger level. If we chose to follow the name that shall not be named kind of situation, like “Oh my gosh…!”, we would just be chasing them, and not going after what we feel is important, and what we think is more fun. As best as we can be, we are in control, while also being in a world of chaos, and total change every day. Right?
I think we’re in such a similar situation…
That’s where we were as of this morning. As of this morning, we decided to put our efforts into ensuring that the legacy and the process of us maintaining anything… If there was a tragedy, something befalls us, that our customers are in a good place, that things are in a nice, responsive little box that people aren’t left in the lurch. That’s what we decided this morning.
How much – I’ve gotta ask you this too, just because it’s such an analog for me, personally. I’ve watched this over and over, and I’m seeing things that I haven’t seen before. Have you all watched - I’m sorry, Jerod - Silicon Valley end to end, like all the seasons?
I’ve never seen a single episode.
Neither have I.
So we’re on Jerod’s side.
My people, my people.
Yeah, I was prepared for this.
I’m sorry…
Not your people… Okay, so you’re missing out on wisdom here, okay?
Are we?
Oh, yeah. I mean, it might hurt you a little bit, but I think there’s wisdom there. And I won’t spoil the story, except for that you may assume that this name that shall not be named - if you want me to keep saying that, I can.
No. Adam, you have carte blanche. You can do what you want.
Well, hatchet.run then, I suppose. Hatchet is who you’ve considered there. They’re a Y Combinator – and it’s unclear on how well they’re doing. Like, you can assume that from the outside. But on the inside, they could be not doing well, and not really competition.
We saw it as just kind of like a counter data point, of like “Oh, that’s what that looks like. If you do it like that, that’s…”
And I think their use cases are great. There were lessons that we learned in going through their material. We don’t have fear in our hearts about them, okay? We’re not expecting them to pull the sword from the stone… None of this.
No fear at all.
[00:48:07.19] Okay. Well, you were just saying how you were making sure your customers were made good if something happened, a tragedy… So you were speaking melodramatic about the potential.
Oh, she’s talking like really tragic. Very tragic.
Oh, no, no. I mean –
Yes. Bus factor of two.
Of two. That’s all I’m saying.
I understand. I was confused.
Sorry, I’ll clarify. We are riding around Scandinavia, Europe, cars… I mean, sometimes we take – we don’t take super-risky decisions, but we’re adventurous people. It’s good to have other contingency things in place for people who depend upon us. That’s all I meant “Our customers will be in a good position.”
Gotcha.
That’s what we’ve decided. Because you could take a deep-dive into our business, and our time management, and everything that we took so long in the beginning to set up, I feel like we’re in such a good place with it now.
Can you describe your business? Can we go to like your pricing page and grok what you might be doing? What’s the best way to examine from a business level? Like, beyond the software, beyond its ecosystem where it hangs out as, as a business, how do you work?
Well, you invited this, and I have to answer…
What’s your business model?
Our business model is open core, in essence.
Yeah, totally.
We sell a couple of paid packages, or a bundle of paid packages on top of an open source offering. So the two packages - one is web, which is a LiveView-powered dashboard that people run right in their main application, that gives them control, and access, and metrics, and stuff, all to their interior Oban working. And then we sell Pro, which is a bunch of more complex, more advanced features built on top of Oban, that is all possible through extensibility. And then with either of those, we have a metrics package, and we have support.
And there’s a custom Enterprise plan if somebody reaches out.
And then there’s a custom Enterprise plan, which is if people need more support.
Okay.
And that’s the plan. That’s the whole thing.
Break: [00:50:13.04]
And what was your freedom number, and when did you reach your freedom number?
What was the first one?
What’s your freedom number? I was gonna go there too, because we’re now back to understanding more clearly your freedom number, what does it take to get there, how close are you to it, etc.
I feel that I have to answer this, since when I messaged Jerod most recently, I poked him with this.
You did.
It’s just – I don’t like specific numbers.
We’ll use percentages, and things.
Use a range, or something. Yeah, don’t be–
Adam’s right. Don’t be gratuitous.
Yeah, the salary of a senior developer in the US. A very senior developer in the US. That would be the freedom number.
Okay, so half a million or more.
Below that. Let’s say between a quarter and –
I was sitting with the wrong people then last night. Gosh, somebody’s –
You’re with some Google/Apple [unintelligible 00:54:06.19]
Yeah. I mean, Silicon Valley people.
In the FAANGS.
Chicago. Chicago kind of people.
Non-FAANG developer salary. So we’re talking –
Yeah, non-FAANG developer salary.
200k to 400k, right?
Correct, Jerod.
Okay, so that’s your freedom number.
Correct.
Yeah. And then we’re over 200% of our freedom number.
Well, congrats. That’s awesome.
Well, that’s a good place to be.
So now you need to go to Mojito Island. Isn’t that where you go after you reach that?
That’s right.
[unintelligible 00:54:31.23] servers…
There you go.
Yeah, you’re on your sailboat, and I’m on the beach, and I say “Looking good, Shannon”, and you say “Feeling
good, Parker.”
“Feeling good, Parker!” [laughter]
Yeah, I wouldn’t be too nervous about your existential business at this point… It’s good.
Yeah, so it’s not an existential thing. It’s just where do you want to –
Where do you want to spend your time thinking about. Do you want to grow into this other ecosystem, where there’s competition, or do you want to firm up your foundation and make sure that if you do go out to Mojito Island and forget to come back to work the next day, then it’s gonna continue. Something’s gonna happen.
Right.
I think there’s a there’s another part, which is this is like the acolyte in me, of I think Elixir as a language and as an ecosystem is just better than any - not every other language, because of course, it’s not comparable to writing C, or Rust, or Zig, or all these other things, that just have a different category they fulfill. But if you’re writing web technology, I just don’t think – this is my total hot take; I just don’t think anybody should be writing Next.js instead. I don’t think anybody should be writing Rails instead. I think there’s just more bang for your buck that’s there. You’ll have fewer servers…
Well, there’s your answer then. It’s just content creation. Why Elixir is better than Next.js. Why Elixir is better than Rails. Obviously, these are two different things, but you get the point. And then you just create content and you grow the pool of Elixirists. And then your business grows, if you’re fine with growing a little bit slower, a little more tried and true… Unless, you know, vertical hockey stick, then you just work on the overall Elixir ecosystem, and as it goes, Oban goes.
Right. Be the change you want to see out there, right?
Yeah.
If you want the market to grow for the camp you want to hang out in, and not ever write JavaScript again, like you said, then you’ve got to put some insurance in place, and the best insurance is your wisdom, and directing folks to the right – to what you think is the way.
You should do some business counseling, Adam. You’ve got some wisdom there.
We do it, as a podcast. Come on back.
This is it, man.
This is it.
Yeah, this is how we do it.
You’re getting it.
Well, we’re getting people on here sometimes, Jerod, and they leave pretty pumped out, like “Man, I love what I do now after talking to you two.” Oh, yeah. Well, that’s part of the game, too. We’re users of Oban. We’ve been users since way back in the day, I think as back in the day as we can go, when Jerod was like “Hey, we need to deal with these background jobs, and queues, and stuff like that.” I think it was like email stuff was happening, and…
It wasn’t even me. It was Alex Koutmos who brought it in the first time.
Right.
And I said “Hey, I haven’t needed background jobs stuff like this before.” And we needed it because I think there was a specific use case… So we had Alex doing some contract work for us. I think it was when we decided we wanted people to be able to edit comments for like three minutes, or five minutes after you posted… Just the typo fix. And so when you send a comment on one of our episodes, everybody who’s on the episode and hasn’t opted out, or whoever you’re applying to, they get emails. So there’s logic there about who gets notified. Our comments system is nice, because it’s not heavily used, and so each comments pretty high signal… And you want to know about it; there’s very few, so you get the email. No big deal. However, we want to give the people the opportunity to fix the typos before we send that email. We’ll put the comment content in the email notifications; you don’t have to come back to the website, you can read what they said.
[00:58:15.28] And I said “Alex, we want to be able to edit those comments, just because typos. We just want to be better than Twitter.” This is back when you didn’t have edit support. I’m like “We’ve gotta have edit support, because we’ve got to be better than Twitter.” So he went ahead and was working on that, and what that required was basically a future-scheduled background job, that when you create the comment for the first time, it’s gonna schedule the notification to go out in five minutes, or three minutes, or whatever the threshold is. And we couldn’t get that done without Oban. I mean, we could have done it some other way, but he liked Oban, so he reached for open. And then I was like “Well, now that it’s in here, we might as well just use it for all the other things that I was just using processes”, for spawn, on or whatever it is. I can’t remember the actual module name. I’m sure you know it, Parker.
Task? I’m pretty sure it’s Task.
Yeah, exactly.
I mean, you’re still using Task. You’re just using Task –
Yeah, I still use it.
[unintelligible 00:59:06.21] by other things.
Yeah, exactly. And so I was like “Well, let’s just put everything in background, now that we have the library in here, and we’ve already got the Postgres table set up…” So we went through – and I think you did a lot of that work, Parker, didn’t you? You came in and opened up a pull request –
Yeah, big time.
…which was helping us use Oban better. And so that’s another way to do adoption, is just one open source repo at a time, go in there and open a PR…
There are a handful of really prominent open source Elixir applications. The Changelog is one of them, but like Plausible is another really big one. And Papercups, and there are a few. I’d like to mention they all use Oban, but in different ways. But any one of those should, hopefully, because people go there to learn “How do I put these things together? What are the popular packages?”, they should be idiomatic. It should be kind of how you want them used.
Yeah, I think that was a really good use of your time, going ahead to open up that pull request and making sure our use of it is idiomatic… Because while we do not use Phoenix idiomatically anymore, as I stated earlier, with context and whatnot, and LiveView, we still have controllers in our code. I know those are out of fashion now in Phoenix land… Our use of Oban is idiomatic.
They’re legal. You’re allowed to do that stuff. It’s fine.
Yeah. Like I said, our app still compiles and runs, so… But our Oban use is on point. So if you want an example of Oban use - oh, yeah.
Yeah. You’ve even got a bespoke feature added to web for your bizarro authentications game.
Oh, yeah. Remind me what I requested, and why you built it for me.
I think it’s something – I know what the feature is, I don’t know exactly why you guys were doing it. It’s a way to have authentication where it just redirects somewhere else. So if you’re not staff, or not admin or something, you can’t get in, but there are certain people that have access who aren’t necessarily you… So you want them to have read-only access. And I guess it was like three tiers of access, really.
Yeah, it was just a case of – yeah, exactly. It was beyond simple kind of boolean in or out. It was like three tiers. And I can’t remember the setup. I can look at the code and have my memory jogged, but it was basically slightly more complicated auth for the web access. And hopefully, it wasn’t too much of work for you, because it was –
It was months. Months of work. That’s all we did, for months.
[laughs]
And you don’t even remember.
I don’t. I mean, I remember being happy when it landed, and I cut over to it, I’m like “Yeah, it works now!” Because I was doing something funky to work around it.
I want to give you some praise, though. This PR was awesome, because I think it showed us early, and I suppose even to now, what your intentions were with the level of support and care for the software you’re creating.
It’s so dedicated. It’s obscenely dedicated.
Well, I don’t know about obscene…
It’s obscene.
Well, I had to even email you. I recall, I was so –
It’s responsive.
Yeah, I was just like “I cannot believe that you did this.” I think I’m gonna read my email, to some degree. I’ll at least paraphrase some of it, or literally quote it. I was like “First of all…” Like, how do you begin an email with “First of all”?
It sounds like we’re in trouble.
Oh, no…
[01:02:10.20] I’m not threatening you here! No, I’m not! I am thanking you, okay? Thank you for that awesome PR. I went looking for your GitHub sponsor page, and after that PR, and found the Oban site, and Web Plus Pro, we’d love to support the development… And I think we supported for a bit, and then we became Pro users, and something like that. And so I was like – I couldn’t believe this, that you had gone so deep on this PR, and helped us in that way. Because like Jerod said, we didn’t know how to use your tech, really, and you came in like “Here’s how you use my tech.” And that’s how we use your tech, and it’s awesome.
You’d think just writing how to use something would be a better approach, but sometimes words are hard, and it’s easier to put [unintelligible 01:02:50.13]
It’s true. Sending links to docs doesn’t always give the best –
That’s true.
It’s a personal touch. I think that the world needs more of that, the personal touch.
You get one of us. That’s just the way that it lands.
Right, because there’s no one else to get. It’s gonna be one of you guys.
That’s right. Another ranty kind of thing, but I just wish there were a lot more independent shops out there. It doesn’t have to just be two people; just a few people. There are some mega software companies. I just wish there were a few just – because the personality is great. It’s like when you live in an area, and you know the people that own the restaurant, or you know the people that own whatever the shop is, and you have that rapport, and you know exactly what you’re gonna get, and when there’s something wrong, you just have a person to talk to.
And you made a difference, Adam. I remember where we were sitting when he opened your email, when you responded.
Is that right?
Yeah. So this happens all the time - where you write something, where you get an accolade, or you get a response back from somebody, or where you’re working on something, we remember where we’re at all the time. People meld to these places. So you made an impression with him…
[01:04:06.02] Saying thank you has gotta be part of it, too. People don’t say thank you often enough. They just sort of accept it and move along. I don’t know if they’re shy, or just maybe they feel like they might be bothering… I don’t know what makes somebody not say thank you, but… I was moved, and I was like “I’ve gotta email these folks and tell them hank you.” And at the time, we were really trying to do a lot more with - and we still are, of course, but at the time we were becoming more aware of like the tooling we were using, the dev tooling we were using, and trying to give back on GitHub Sponsors, because that was becoming more and more of a thing to do… And we obviously wanted to support you all in your efforts with Oban.
How do you feel about cron jobs? Do you like them? Do you hate them? How does this kind of merge into your world? Because I want to give a shout out to somebody that I love, and they’re pretty awesome, but they’re in the cron world, and I don’t want to like ruffle your feathers if I don’t have to.
Well, so we have cron as part of Oban, and it’s purposefully kind of bare bones. So you have a static crontab that you set up, and when your app starts, it’ll go by that. It’s got some niceties, like the reboot keyword, and some of the things like that. But then in Pro, we also have what is dynamic cron. And that has some things like guaranteed scheduling. So if you were supposed to schedule something like at midnight, and it’s only once a month, but your server happened to restart or something happened, it will go back and identify that you missed that. It will also let you update crontabs…
So retries and misses.
Yeah. It’ll let you update them at runtime, or
you can pause things. You can pause certain jobs, and you can insert new ones. So we feel pretty strongly that cron’s an important part of every production app we know of. Because we get a lot of config that people send us, for like diagnosing…
Well, that’s where it began for Jerod, too. He’s like “I was using cron for these things”, and whatnot. So…
Yeah. But there’s also the benefit that we – because it’s Postgres-backed; it’s like it’s centralized. And anytime you have a setup of two nodes, now you have a distributed system. And either they have to have consensus, or you have to centralize through something. And so we kind of get away with a lot because of that centralization part. But I’m curious who your cron friend is.
Well, yeah, I’ll mention them, but I want to go on the note where you say you wish there were more - was the word you used small shops, or small teams? How did you say it?
Yeah. Independent. Not faceless mega corporation kind of places.
I think they get kind of hidden. And the reason why I’m gonna say these folks is because they were kind of hidden to me until I had the problem… And so it’s not in the application world, it’s more at the Linux level world. So the app I’m gonna mention is called Cronitor. Shane and his team - they’re going to become a sponsor, because I’ve been such a fan, and I’ve been like hunting him down, telling him how I can help him… And I’m just like a super-fan of cronitor. So the website is cronitor, and I use it heavily. And there’s lots of folks in the home lab world that I think would use it more heavily, because there’s so many cron tabs users out there, and cron just fails. I mean, you can go read a log file, but that’s kind of boring, right? You want it to go to an interface; it’s a little easier. And you want retries, and grace periods, and more sophisticated things, which is all part of the Cronitor platform that they have.
And another name that comes to mind is molehill. Now Molehill was an early days Ruby shop. Jerod, you may remember this, they’re from the Florida area. This is like 15 years ago. Well, you may know what they do now, which was they did some things in the nonprofit world helping applications out, but now their application is called Buzzsprout. And you may know that, Jerod, because they are deeply into podcasting. And that was where Founders Talk, I think, and the Web 2.0 Show, the earlier podcast I had done, and I think the Changelog was hosted there before we… Did we move that from there? I’m pretty sure we did, Jerod… Before went we to 5by5, and then from 5by5 to our application.
[01:08:00.01] So Molehill was an early interface application development company, small shop, independent shop, however you want to frame it… Cronitor is very much the same. They have amazing revenue. They’re a few people. The CEO is doing support, kind of thing, answering my emails, and eventually sponsoring our podcast… I want to mention that, because – I hunted them down. I’m not mentioning them because they’re a sponsor. I was so emphatically like “Y’all are amazing. We’ve got to talk.” And then over a period of probably eight months, we finally pinned something down to work together. And I’m just a fan of small teams like that. I agree, I think indie teams – and I wonder why…? Is it because of cloud? Is it because of the behemoths out there? Do they just get overshadowed? What is it that makes indie teams not – I don’t know even how to describe how they’re not there. Or are they just hidden, and they’re hard to find? Because I didn’t know about Cronitor for a while, but they’ve been in place for years. Successful enterprise teams. They’re a small team with really big businesses using them.
I can hypothesize. I don’t have any –
Please do.
This is all anecdotal, but I think safety’s one part of it. It’s hard to be a contractor; it’s really hard to start software from nothing. To bootstrap something is tricky. And the amount of time it takes, if you have a day job, to build something else for that safety, to try to get to that place where you can actually run it without worrying about an income stream - it takes a lot of dedication. And that’s double work, in some ways, and not everybody’s up for that.
But then also, you get so many startups, and what’s the goal of most startups? I mean, they’re incubated in some way; they have people backing them, therefore those people want a payout. And usually, you have to grow to be able to get there, or you’re letting down your investors. And if you let down the investors and you don’t get to some sort of exit, then as a business you’re kind of a failure.
Whereas if you’re an indie mom and pop kind of place, or however you break it down, even if it’s just a few employees, you don’t have those – you just don’t have that incentive. It’s your lifestyle. And so I don’t know if people see lifestyle business as a detractor, but to me it seems like –
Possibly.
…the way I would hope more people could live.
He’s an evangelist.
You can speak to this, Jerod, a little bit, right? You can speak to this, Jerod, because you ran Object Lateral for multiple years as a solo dev, and you always seemed to me… I can probably suggest some ways I thought you felt, and to some degree, but you always seemed very confident in your ability to… And you were always very customer-focused. He mentioned even recently, “Hey, I’ve had this responsibility with this friend of mine. I no longer do day to day work with them, but I had to help them spin it up.” And we were lamenting how hard it was to get their stack back into place, basically.
Right, right.
But you’ve had – as Object Lateral you were an indie, individual person; that was your full livelihood, you raised a family on it. Obviously, now you’re working with Changelog and that’s in your past… But for a long time, you were just helping people, and slaying apps, and hiring contractors, and getting the job done.
Yeah. I agree with all that. Was there a question in there or something you wanted me to respond to?
No real question, more just like share your experience.
Share my experience…
Yeah. Well, on the idea of indie, mom/pop, the fear, the lifestyle… And I think what Parker was hypothesizing was just essentially the fear of like not being able to have this big job, with a big salary, and whatever.
Yeah. Well, I think that growth mentality is just [unintelligible 01:11:35.13] and pervasive on the internet, probably because of Silicon Valley, the ethos, not the TV show.
Both.
Do you get a chime for that one, if it’s just the name, not the place?
Every single time. Every single time.
I specified it’s just the ethos. I did not reference a television show. So no chime.
[01:11:56.06] I said both, so there’s a chime.
I have a question. Do you feel that the lifestyle, mom and pop, what we do – we’re in a bubble. So when we break out of this bubble, and occasionally attend a conference, or occasionally go to a meetup, do we drive people away from choosing us based on how intrinsically our lives are wrapped around our project? Are we driving them away?
I don’t like the way you’re phrasing this question…
You personally, just the two of you are just turning everybody off personally to that idea of it.
Just everybody’s running –
I don’t think so, no. I think people will say lifestyle business. And when they say that, it’s a bit of a pejorative. They’re like “Oh, it’s a lifestyle business. Isn’t that cute?”
Yeah. I don’t think it’s right. I think it’s wrong.
I feel that term is perhaps detracting.
Right, perhaps. I think we need to take it back. We should take it back. But I think that what I did and what you guys are doing is slightly different. Well, more than slightly.
Very much, yeah.
I think it’s different because I was effectively a dev shop. I was a glorified freelancer. There’s a lot of freelancers making livings on the internet, writing software on contract… Service companies, like building software, whether as a service, or products like Oban Pro, and selling them to customers - I think it’s a little bit of a different beast. And so I think there are a lot – there’s a very vibrant freelance community, and some people call themselves Dev shops, or contractors, other people say they freelance… Just like some people are developers, other people are programmers… But I think that’s alive and well. And I think there’s more of that going on, maybe than ever. But in terms of people starting small software companies in order to sell software, whether it’s as a service, or as a download, or whatever it is, there’s definitely a lot less of those that stay small, or start small for this, for that matter.
Well, we have a few things going for us. We started later, rather than earlier, which is great for us. We had already tried a few other endeavors. I feel like that made a huge difference for us. I also feel like – we stopped consulting. I realized that you have what they refer to as a day job, but that’s really the only thing now, aside from Oban. We were consulting with about two clients each. So I think before COVID, we were - I would say we were pressed for time beyond compare.
Yeah.
So we stopped consulting and taking on clients. And that was right before COVID really kicked off.
My mind jumped to a couple of things too, because if you’re running your own software shop, or your own product, you have to wear so many different hats. I mean, there’s the entrepreneurial spirit itself. And then it comes down to marketing, and then having the technical chops to do that, and then having the support side, and then having the writing… There’s just so many things you have to do; balancing all that is difficult by itself, especially if you’re not bringing in a lot of other contractors. And I think the language you choose can make a difference as well. Some things - it’s just, it takes more time; it’s harder to do something with a couple of senior people in some ecosystems, compared to others.
Yeah, there’s no one size fits all. I do think that maybe when it comes to software products, it may be that the successful ones, because of the profit margins on software sales, which you all are enjoying… Like, every marginal sale for you has not nowhere near as commensurate work on your side, right? You have support, but you could just sell them to the hills. And so when you start to actually get over that hump, and the flywheel is rolling - well, it’s hard to stay small at that point, because…
[01:15:53.27] The other one I’m thinking of is – and they actually sell hardware, so it’s slightly different, but it’s similar… It’s Thinkst, with their canaries. We had Haroon Meer from Thinkst on the show last year, talking about Canary… And they’re relatively small, and making really good money selling these security devices into enterprises. They’re like honey pots, basically. And they have a hardware aspect, which means that their margins aren’t as good. But once their sales got rolling, they’re making really good money. And he has to try hard to stay small, because it’s so easy to chase every opportunity when you have capital to spend.
And so maybe that’s why people end up chasing larger growth, is because, once you’ve conquered the mountain, then you’re like – wasn’t there an old saying about a guy who cried because he had no more mountains to conquer? Was it Alexander? I think it was Alexander.
Come on, Jerod…
Do you guys know that one?
I don’t know that one, but I like the idea.
Alexander as in the Great, or…?
Yeah, yeah.
[unintelligible 01:16:55.23] or which Alexander?
[laughs] The Great, of course. Anytime you leave off the second part, it has to be Alexander the Great. Who else could it possibly be?
The Terrible?
That’s Ivan.
That’s Ivan the Terrible. Yeah.
Our growth has been stable. I feel great about it. I really don’t know what I would say other than our growth has been stable, and I feel like our churn is kept relatively low… All of these things are great. When we go to balance our time, Parker’s right though. It will be Monday for writing its blogs, it’s Tuesday – if something’s on fire, and we get a request, then that takes immediate priority. But we really do have to stay on top of – there’s another book I read, years ago though, about the pomodoro. Like, your small tasks, all day long… It seems like we run these little box checklists with each other. It is not uncommon for the two of us to sit right next to each other and not talk for three hours. Which - he’s like “This is a lie.” He’s like “I never get three hours.”
I think this is a complete lie. I don’t think this has ever happened.
He’s like “I never get three full hours. Somebody always –”
“One of us is talking.” Yeah.
Well, if we get an email from Adam, or from Jerod, then…
You’ve gotta pause and read it.
Then you high five each other. Do you guys have any sort of celebration routines, like when a new sale comes in? A lot of people have some sort of a bell that goes off, or a thing on the wall… Do you guys do high fives? Do you give each other a hug?
Of course we do.
In the early days, we did. We would celebrate with steak, especially if we got like an annual sale, or something. But you can only eat so much steak. I mean… I’m sorry, I know there’s a Texan available… But still.
Which brings me back to my Alexander the Great quote. Here’s my Alexander the Great quote, which - you can only eat so much steak… [laughter] I don’t think he actually said that, but the quote is “When Alexander saw the breadth of his domain, he wept, for there were no more worlds to conquer.” That was from Plutarch. I think there’s arguments about whether it was actually from Plutarch, or whatever. But that’s the idea.
He needed a pair of binos. He needed some binoculars.
No, so [unintelligible 01:19:11.04] He’s DHH, in a way.
How so?
He’s DHH. Creates Rails, conquers the world. Basecamp does amazingly well. He starts a racing career.
He’s Formula 1.
Does Le Mans. Everything that he attempts works perfectly. He conquers it, and then he’s bored. And then he laments, like “What am I gonna do next…?” It’s just, you keep conquering all the little mountains, and you get there, and then you’re bored. And maybe I’m a lesser person, and maybe a lot of us are lesser people, but – like, I just redid a guide today. And that guide - it’s clear, and it helped these other people, and that’s like a small win… But that’s enough to just keep this constant forward momentum. So you’re smoothing out the mountains. Instead of climbing a mountain and then finding a bigger mountain, you’re just kind of constantly moving upward, and finding some joy and peace in it.
[01:20:09.02] Right.
The zen.
Sorry, DHH.
I’m over here thinking DHH is – you know, when he gets bored, he goes on Twitter and throws Molotov cocktails. He’s bored, so he’s gonna go get into verbal spats with all of humanity.
And what an interesting person though, really. He is a caricature for our – I mean, innovative, leader in a lot of cases, but also just like a unique individual, that’s for dang sure, man.
If he’s looking for any ideas, I feel costuming and learning to fly commercial airplanes is – I think he should go that route.
Both?
Costuming?
One, or the other, or both?
Put on maybe some –
I think that’s Mitchell Hashimoto’s territory. Flying small airplanes.
With the airplanes, that’s what I was thinking of.
Yeah. Well, when you have the mils, you can do the things. And you can get cars and drive Le Mans, which is, you know…
I think if Alexander the Great had more hobbies… Even available. Like, Bill Burr - he flies a helicopter. And I feel like if Alexander –
The comedian?
Yeah. He’s a trained helicopter pilot, and he –
He seems to have so much rage for flying a helicopter… I don’t know if I’d trust that.
I think he only flies by himself for the most part, but… That’s one of his hobbies. He does that to decompress. And I feel like if Alexander the Great just had like a helicopter, or an F1 car, or something…
A mountain bike.
…he wouldn’t have cried so much. He wouldn’t have been such a crybaby.
Well, I can agree with that. I mean, gosh, I’ve been so at peace in my garage, just turning a wrench on my mountain bike lately… I’ve been doing some cool stuff, and I’m just like – it’s my happy place. I put some music on, I jam for a couple hours by myself… No drinks, just turn a wrench, and it’s good. It’s analog. It’s not a computer, it’s not a screen… My son would come out and help me for a bit, too…
The calm balm…
Oh, yeah.
[unintelligible 01:22:00.05]
Just calm balm. Just the essentials.
Is that [unintelligible 01:22:04.20]
It’s 300 milligrams. It should be 600 to really take me to where I want to go…
[laughs] Oh, gosh…
Yeah. Well, we’ve got to find our peace in our hobbies, that’s for sure. But I like your idea though of flattening out the mountains though, because it’s a good thing. It’s a good thing.
There’s a lot of zen to our life. There’s a lot of balance. Sometimes I feel like –
I think our children, who are here - there’s a lot of zen to our life, when they listen to this, they are like “What are you
talking about?”
From the chaos.
Yeah. When it’s chaotic enough, it just averages out.
Well, even with this pivot you said earlier, and this non-incumbent coming in, making you think differently… Big change. I said this before, Jerod, big change brings big change. We don’t always know the ripple effect of a new decision. And if you like where you’re at currently, and you like your relationship, and that’s where it’s at, I would fight to stay small. And not small as in like not grand vision, but small as in like that you’re still in charge, that you’re still calling the shots, that you can maintain. Things that Jerod and I established as rudders to our business has been “Slow and steady wins the race.” That’s a pretty common thing for folks to think about, but I don’t think that Slow and Steady means you literally go slow. I think it means you can go as fast as you can, while maintaining a version of control for where you’re trying to go. That you’re aware of where you’re trying to go, and you’re going at a speed that is comfortable. That might be 100 miles an hour. It doesn’t have to be necessarily slow. It’s more like slow and steady. So it’s the two together, it’s a pairing.
[01:23:50.24] The other one is if we get sort of over our skis, and we’re sort of like rethinking like “Oh my gosh, this is a little out of whack”, or things are off kilter, “Slow down and check yourself.” What are you optimizing for? Those three things, like “What are you optimizing for?”, “Slow and steady”, “Slow down and check yourself” have been lifesavers in my personal life and in the business Jerod and I run together. And I can imagine he probably feels the same, because he does say so. So take that as you’d like, and apply it as you’d like, because that’s helped us.
Yeah, I think you get great ideas in the spur of the moment, but anytime you just take that, manifest it and then push it out into the world - it wasn’t quite there. Like, you just have to take a step back and look at something. In art school, you’re taught, when you’re drawing or you’re painting, “Take 10 feet back, look at it again, and then come back in and rework it”, because until you take some time away, you’re just not going to see everything.
Perspective. That’s what it’s about.
Right. You need time for perspective.
Yeah, for sure.
You can’t borrow it. Nobody can give it to you. You have to build it and grow it.
And Adam, I think you said that emotional response you have - that’s not always the one you want to lead with. That’s not the one you make the decision with. It’s important to acknowledge it, have that conversation…
Yeah. And even have it, too. I love – I mean, I’ve written some messages that I did not send, okay? But it felt good to write the message.
Is that what we got? No. He did send it. [laughter]
He did send it. [laughs] In reference to us discussing that pivot or not, we ran the tracks on a lot of different versions of which way we could go. And emotionally, really hearing each other. I can think of a house we went and looked at when we were first buying homes, when we were home shopping, and Parker turned to me. The minute – I mean, we had already put an offer on this house, and it was during the home inspection. And he turns to me, and he’s like “I don’t want this house. I don’t want this house.” He’s like “I don’t want this house. This house will cause us to fight.”
I can’t stand in the shower. I can’t stand in the basement. It’s got black holes in the ceiling. It’s too small.
He just had this instant tirade, and –
It’s too small? Are you a tall guy, or what do you mean you can’t stand in the shower?
I’m not tall.
He’s a behemoth.
I’m not that tall. I’m over six feet tall. I’m six foot two.
Okay. You have to hunch down in the shower?
It was a submarine shower. I don’t know what was going on with this thing.
It wasn’t. It was like an 1860 farmhouse, and…
Okay, I get that.
…in a commuter town outside of Chicago. So you know, a wraparound porch… I’m not saying this was the dream house, but we had agreed on it, and we were looking through the house, and the inspection is taking place… And I think we’re just kind of wandering around to kill time, and he turns to me and has this – I mean, a very emotional response about “I do not want this house. I do not want what will come with this house. I think it’s going to cause fights. I don’t like the–” And I just said “Okay, well, we’re out.”
I don’t remember talking like I was in a Dr. Seuss book, but I think the sentiment’s there.
He did. [laughter] He said it all – it was a con list, bullet-pointed verbally.
Wow. Did you buy the house?
No. We left. Contacted the attorney and pulled it out.
Have you guys ever seen The Wedding Singer?
Yeah, absolutely.
Remember how his bride doesn’t show up for the wedding?
That’s right, yeah.
And then she comes to him the next day and she’s like “Sorry, I just couldn’t marry you.”
“Once again, things that could have been brought to my attention yesterday…!”
Yeah, I’m just imagining Shannon yelling that to Parker when he’s like “I don’t want this house.” Like, “I appreciate the decisiveness there, and just putting it out there before you make the big mistake… But didn’t you notice that during the walkthrough, Parker? I mean, come on. You couldn’t fit in the shower the previous day, too.”
Did I try to get in the shower the previous day? I don’t know. [laughter]
Try to go more “Place no blame… Let’s just get out of this situation. It’s a dumpster fire now.” I was like “Oh, this is a dumpster fire.” So the first thing – I just called the attorney and said “Yeah, we’ve gotta get out of this.” He says, “No…”
[01:28:10.17] You do things with a partner, whether that’s a spouse, or a business partner, or a friend, that - maybe it doesn’t seem like your idea, but it might work, it might grow on you… And maybe that’s not always a good – you don’t have that fight or flight. You don’t have the flight part. You just know that maybe it’ll work.
So it’s important to listen to the bullet point con lists when somebody decides to give you an honest read like that, and not emotionally – so I feel like we’ve had a lot of that back and forth on whether or not to pivot, and to get to that point of being able to really listen, of like Adam said. Like “Look, if you value this, if this is where your values are, if this is what you’re working towards, recognize that that’s at stake.” I hear you. I hear you, oh wise one.
Big change brings big change.
[laughs] That’s so ominous.
You’re scared.
It’s too calm now.
Can I explain it a little bit?
Oh, I think we understand it.
I wanna explain it. I’m sorry, I have to. Regardless of your answer, I’m explaining it.
We understand it.
Just one second, bear with me here.
[laughs]
Sometimes we make choices in our life, and they are big changes. And we think we’re just making that one big change. And this is coming from my own perspective, so this is like school of hard knocks, bloody knuckles situation. And you think you’re – okay, you’ve risk-factored that single one big change. And maybe you consider a couple of ripples. But sometimes that big change brings other big change that you did not derisk. And that’s where my perspective is with this. Sometimes big change brings big change, because you’ve thought through the one thing, but have you thought through all the permutations? Now, we can’t do that, but everyone knows what a version of big change is in their life, and apply it accordingly. Big change can bring big change.
So beware of the contagion.
Yeah, I mean, there’s ripple effects for sure. The one big change you are considering and saying yes to isn’t the only big change possibility as a result of the Yes. And likewise, the No.
I feel like that’s potentially crippling advice, though.
Sorry, Jerod.
I mean, it just ominous. You’re like “Don’t make a big decision, because it’s gonna cause other big ripple effects.”
I didn’t say don’t. I just said big change brings big change. Prepare. Take a towel.
Yeah, but… Is that not a warning?
It’s a warning to consider your big changes. Yeah, I think it is a – I don’t consider it a warning. It’s more like a red flag. I guess that’s a warning, I don’t know… It’s not like a negative warning. It’s just more like “Hey, if you’re making a big change, consider there’s other big change you haven’t considered.” And don’t go in with zero margin. Have some margin. When you go into a room, plan your exit. Or consider that you might need to exit.
Or when you go to buy a house, consider an exit, you know?
Yeah, I mean, I think that’s just how I operate. At least I try to… And I get really upset when I give advice, and I don’t always take my own advice. And I’m like “Adam, you knew better. What is wrong with you?” And then I podcast about it. That’s how it works.
Oh, not taking your own advice is the life of –
It’s the worst, man.
…every senior developer.
It’s true.
Yeah, it’s the worst. It’s the worst when your own advice laughs in the face, like “You knew better. Come on.” But that’s it. Big change brings big change.
Well, we are excited for you two. We are excited to see what comes next, what big change, your next big change is going to bring… I’m looking forward to some new Elixir content, some hot takes from both of you. Some widening of the Elixir ecosystem. I would love for you to put on paper or perhaps on video how Oban is a killer app that should make you pick Elixir. Exercise that thought all the way to its logical conclusion, and make that argument. I would certainly read that, and I would certainly share that around. I’m interested in that. And congrats on all the success. I mean, you’ve doubled your freedom number, and you’re still rocking and rolling.
[01:32:14.15] I just don’t believe it will ever lead to a retirement… And that’s great. Because that means –
You have options now.
We have options. And the freedom is there. It’s just the freedom to choose.
The reason that people retire and then they end up going and getting another job is that they have no more mountains to conquer. You’ve just gotta keep it going.
That’s right. It always goes back to Alexander the Great.
Smooth the mountains.
So this is what Paul Vixie said to us last week… I asked him, because Paul Vixie has conquered many mountains, so he’s an internet Hall of Famer, instrumental in the DNS protocol etc. He wrote Bind… Actually, Vixie’s cron is still the cron that runs in most Linuxes today. His implementation of Cron is still there. So he’s been in the business 40 years, 60 years old, working at AWS at some high level… And I asked him why he hasn’t retired yet. And he said “What I learned about myself–” Because he had an opportunity to kind of call it quits, he sold another business a few years ago… And he said, “What I realized is if I didn’t have a reason to get out of bed, I’m not going to get out of bed.” And he learned that about himself. And so he said “So I went back to work. And now I have colleagues, and I have projects, and I have trips to take, and stuff to do, and I’m just a happier person with a job than I was without one.”
Not that he’s every person. Some people can retire and find all kinds of stuff to do for themselves that are not money-producing… But I think we do need to have a purpose and a reason to get out of bed in the morning. Otherwise, we just kind of waste away, and… I mean, heck, some people die young because they retire young. So that sucks.
You’ve gotta make something or fix something, or the day is just not complete… So I hear him on that.
Physical or digital.
Well, and I’ll answer your call. Content is coming. And we have a talk at Elixir Conf EU in Lisbon, which is about scaling applications, and we have an application that we’re building to push the boundaries of what’s kind of possible for job processing… And that’ll be part of that talk, too. So we’re working on it.
We’ll see if we can get in the “How Oban is a killer app for Elixir” hot take before Elixir Conf EU.
I love it. There you go. When you write that, or record that, or whatever turns out being as media, let us know.
It’s been fun. Bye, friends.
Bye…!
Our transcripts are open source on GitHub. Improvements are welcome. 💚