Blog

88: The Same Bucket

The garbage truck that exploded outside my house means I am karmically bound to share a piece of the networking book.

Authoritative nameservers contain the information for one or more specific domains. I run authoritative DNS servers for my domains, such as mwl.io and prohibitionorcs.com. Anyone in the world who performs DNS queries on my domains gets their authoritative answers only from my servers.

Recursive nameservers provide DNS lookups for clients. When you browse to my web site your computer asks a recursive nameserver for the IP address to connect to. The recursive nameserver finds the authoritative nameserver for the destination site, queries it, and returns the answer to your computer.

Put your authoritative and recursive nameservers on different hosts. The twentieth-century practice of combining authoritative and recursive DNS on one server led to many security problems. In hindsight, the “store the sacrosanct Single Source of Truth for our company’s public face” function and the “collect and cache random data from any system anywhere on the Internet” function should not share the same bucket.

Remember folks, don’t throw lithium-ion batteries in the trash! But do sponsor Networking for System Administrators.

Posted on

Migrating from Apache 2.4 to Caddy

I’ve been using Apache since the 1990s. The networking book requires information about QUIC, so I need experience with QUIC, so I need HTTP/3, so I can’t use Apache.

I experimented with Caddy on my test host. It worked well as a reverse proxy, so I began putting it in place in production this weekend. (If you deploy Caddy, definitely have it run as a user other than root!)

As I went through the docs to prepare, though, I realized that not only would it would be less complex and more robust to drop Apache and use Caddy, it would also be easier.

My Apache configuration files are large and complex because Apache can do anything. I don’t need a web server that can do anything. I need a web server that serves static files, talks to php-fpm, and supports TLS. The Caddy docs are complete, but I didn’t find a simple guide for what I wanted to do, so I’m posting this. I suspect that guide exists but is buried beneath pages of search engine poison.

This uses Caddy 2.9.1 on FreeBSD 14. My config files are in /usr/local/etc/caddy, symlinked to /etc/caddy.

The main config file, /etc/caddy/Caddyfile contains only:

import sites/*conf

The /etc/caddy/sites directory contains each of my sites in its own file. Mostly.

Here’s one of my old sites in blackhelicopters.org.conf:

blackhelicopters.org www.blackhelicopters.org {
        root * /var/www/bh
        file_server

        log {
                output file /var/log/bh/bh-caddy.log
                format json
        }
}

The first entry on the first line is the server’s main name, blackhelicopters.org. (I could probably let that domain go, but my oldest friends have that email in their address books and it’s worth a couple bucks a year to not inconvenience them.) The following hostnames are what Apache would call ServerAlias entries: other names this host responds to. Every name here goes into the X.509 certificate.

The root statement tells Caddy where to find the files for this site. Every URL goes under here. If I had Apache Directory statements, I could put them here.

The file_server statement means “hand out files.”

Last, there’s a logging statement. Caddy logs are written in JSON, making them harder to eyeball but easier to mechanically parse. Pipe the logs through jq(1) to read the parts you want.

Several of my domains exist only as legacy redirects. While http://michaelwlucas.com and http://michaelwarrenlucas.com made sense in a keyboard-centric era, they’re a pain to type on a phone.

blather.michaelwlucas.com www.michaelwarrenlucas.com michaelwarrenlucas.com www.michaelwlucas.com michaelwlucas.com mwlucas.org www.mwlucas.org {
        redir http://mwl.io
        }

This config doesn’t even serve files. It’s like setting DocumentRoot to /var/empty. Any traffic to these hostnames should be redirected to my current web site.

So what about that all-important main web site?

mwl.io  www.mwl.io {
        tls mwl@mwl.io

        log {
                output file /var/log/mwl/io-caddy.log
                format json
        }

        root * /var/www/io
        file_server
        php_fastcgi localhost:9000

        @disallowed {
                path /xmlrpc.php
                path *.sql
                path /wp-content/uploads/*.php
                path *~
        }

        rewrite @disallowed 'index.php'

        redir "/ks" http://www.kickstarter.com/projects/mwlucas/mwls-next-1-april-book"
...
}

the tls statement puts my email address in the Let’s Encrypt certificate request. I should probably go back and add that to the sites I did earlier.

The php_fastcgi option tells Caddy where to find the php-fpm engine.

The @disallowed statement defines a list named “disallowed.” The following rewrite statement transforms requests to files with those names, redirecting them to the index.

Finally, I have several redirect statements for my convenience.

Test a configuration by going to /etc/caddy and running caddy validate, much like apachectl configtest. The configuration files are JSON, so the parser isn’t quite as straightforward as you might expect.

# caddy validate
2025/05/05 15:02:38.489 INFO using adjacent Caddyfile
2025/05/05 15:02:38.489 INFO using config from file {"file": "Caddyfile"}
Error: adapting config using caddyfile: /usr/local/etc/caddy/sites/test-twp.conf:1: unrecognized directive: test.tiltedwindmillpress.com
Did you mean to define a second site? If so, you must use curly braces around each site to separate their configurations.

Here’s the problem: the error is not where it says the error is. The error is before the cited point. The sensible thing to do is to test after creating each site’s configuration file. If you get bored and do all your sites while watching reruns of Adam and Jamie welding JATO units to a hamster ball so they can replicate that urban legend about the Syria-Guam War, you’ll have to do a binary search of your files to see where the problem is. Test each one as you finish it.

Once you have a parseable configuration, shut off Apache and start Caddy. Watch /var/log/caddy/caddy.log for errors. Test all of your sites.

Am I happy with Caddy? Yes, so far. Am I keeping my known-working Apache configuration around? Also yes, so far. If I suffer an attack of the AI scrapers, I might need to fall back to a Caddy reverse proxy so that I can implement Anubis. Yes, there’s an Anubis Caddy module but it’s a proof-of-concept.

What kind of impact has Caddy had on my site? It seems faster, but that might be QUIC aka HTTP/3 rather than any difference between Caddy and Apache. Of course, QUIC is a difference between the two. How much of my traffic is QUIC now? QUIC runs on UDP port 443. First, let’s check how much traffic went to and from port 443 yesterday, on all protocols.

# nfdump -R . -B ip 23.139.82.3 and port 443
...
Summary: total flows: 58605, total bytes: 6.9 G, total packets: 7.0 M, avg bps: 1.0 M, avg pps: 127, avg bpp: 990
Time window: 2025-05-04 00:00:00 - 2025-05-04 23:59:59

6.9 GB. How much of that is UDP?

# nfdump -R . -B ip 23.139.82.3 and port 443 and proto udp | tail -4
Summary: total flows: 1620, total bytes: 428.4 M, total packets: 412444, avg bps: 62756, avg pps: 7, avg bpp: 1038
Time window: 2025-05-04 00:00:00 - 2025-05-04 23:59:59
Total flows processed: 750342, passed: 1620, Blocks skipped: 0, Bytes read: 66537172
Sys: 0.0209s User: 0.0209s Wall: 0.0399s flows/second: 18806544.9 Runtime: 0.0423s

428.4 Mb of my traffic is QUIC? Firefox and Chrome derivatives both use QUIC if available. The only clients that should be using TCP are stupid bots and crawlers–

Oh. Maybe I do need to implement Anubis. Dammit.

Posted on

BSDCan Travel Fund Auction in honor of Mike Karels

Mike Karels has been around the BSD community since the last century, and was integral to our projects. How integral? If your name is on the definitive book on the topic, you’re integral.

On his way home from BSDCan 2024, Mike passed away.

I could go on and on about what a humble guy he was, and how he helped many folks. Or I can tell you that he backed Run Your Own Mail Server. He had no need for my book, but thought it was worthwhile? I was stunned. And appreciative.

With his family’s permission, I am auctioning off his reward in his honor. And something extra.

Here’s a copy of the backers-only edition of RYOMS, Ruin Your Mail By Running It Yourself, with a sponsors-only challenge coin. After fulfilling sponsor gifts, I have a scant handful of coins left. I don’t sell them, despite repeated requests, the occasional threat, and one ham-fisted blackmail attempt. The only way to get one today is by winning this auction.

Bid on the set by leaving a comment on this page.

The auction runs from now until 5PM EDT 12 May. If the bidding goes nuts in the last few minutes, I’ll leave it open until it settles down. There’s no sniping this auction at the last moment, as I want bids to escalate beyond all sensible limits.

Mike was a cool dude. Honor him by giving the next generation a chance to join us.

Posted on

April’s Abjurational Sausage

This post went to Patronizers at the beginning of April, and to the public at the beginning of May. Not a Patronizer? You could be. It’s a terrible deal, but you could be.

I’ve considered myself well-prepared for personal financial disaster, but we’re not headed into personal financial disaster. We’re headed into a global one. I would say that I’m conservative–I save money, look after my family, mind my own business–but the reactionaries have stolen that term from me. Plus, I’m not conservative enough to cope with this economy. Nobody could be.

Last year was an all-time writing income peak, thanks to the Run Your Own Mail Server Kickstarter.

This year? Well, the Laserblasted Kickstarter is concerning. Yes, this is something of a gag book–but it’s a real novel. Folks who would normally take a $200 leather-cased book are backing for $6 ebooks. Several folks wrote me to say that they wanted to support the book, but simply couldn’t right now. I’ve also lost Patronizers.

I don’t blame them. But the upshot is, I’m expecting this year to be ugly.

Plus, this month I’m writing checks for taxes and retirement fund. Because of RYOMS, those checks are large. While I understand the logic of “feed the IRA while the stock market is low,” it definitely feels like I should just set the money on fire instead. The IRS would let me record that as a business loss.

From the latest happenings, I think it’s clear that I need to start increasing my overseas business. I need my books to become more accessible to European and Asian markets. That means dealing with VAT. Even with the RYOMS Kickstarter, I didn’t quite make enough in Europe to qualify for IOSS. I suspect that if I can include VAT in my book prices more Europeans will buy them. If I get an IOSS number I can have my books printed inside the EU, which would make them still more accessible. To get there, I need to sell more.

Chicken, meet egg.

I need to ask folks who run businesses in the EU for their thoughts. Fortunately, there’ll be a bunch at BSDCan.

Anyway. Money sucks.

The good news is, my readers seem to like ordering print books through my store! Each week there’s a handful of print sales from my bookstore. While I still have no idea what my best-selling titles are, I used a couple months of Amazon print sales as a proxy and have started working my way down that list. I’ve set up PAM Mastery, the two versions of TLS Mastery, and FreeBSD Mastery: Jails with the new printer and ordered the proofs. If they come out okay, I’ll have them on the store as print/ebook bundles straight away.

When converting books to the new printer, my titles are either simple or disastrous. I plan to focus on the simple ones first, but ideally I want to offer “The Full Michael” in print. I don’t know that anyone will actually buy that stack, but being able to offer that makes me as independent as possible.

The other good news about direct print sales? Bookvault recently added printers in Australia and Canada. Cracking Canada has long been a goal. Yes, I can ship from the States, but for two countries right next to each other, shipping to Canada sure is expensive and annoying. Printing and shipping from Canada also lets me skip the annoyance of customs.

The funny thing about this exercise is: chapbooks. When Amazon shifted their prices, I stopped putting short stories in print. I might bring short stories into print again, but make them exclusive to my store. If I was smart, I’d put them in a series like William Meikle did. Number them. Make them seem collectible. Brand them like a unit. See if I can trigger that completist urge in collectors.

Oh, wait. My hardcore collectors are mostly Patronizers. I’d just have to ship them copies. Huh. Well, I might do it anyway just for the laughs.

On the writing front: Project IDGAF, aka Laserblasted, survived first readers. People like the stupid thing. Many Patronizers will get a copy as part of their benefits, but if you’re not one of them you might take a look at the first chapter as see if it interests you. First reader reaction was universally positive, which was quite a shock. I wanted to write something that I had no stakes in, that people were not expecting or demanding, so I’d be free to stretch myself and be a little daft. (More daft. Whatever.) It seems to have worked, which is nice.

Which leaves me grinding on the new Networking for Systems Administrators. Writing a cross-platform book is weird. We pretend that TCP/IP is a universal standard, but everybody made different decisions regarding that standard. Sigh. Still, it’s starting to coalesce into something resembling a book.

Hopefully I’ll get N4SA2e to Kickstarter this summer, along with the Twisted Presents Christmas story collection. Most of the stories exist for that one. I still need to write an exclusive Prohibition Orcs tale for it, but I’m hoping that doesn’t go horribly wrong. Last thing I need to do is accidentally write a Christmas novel about a cranky little old lady orc (no matter how much JG would like that).

Anyway, I better get back to these edits. Thank you for your support. I appreciate every one of you, especially in these troubled times.

Posted on

Notes on caddy as QUIC reverse proxy with mac_portacl

As I wrote yesterday, I need QUIC for my web sites. The servers I have data on run FreeBSD, because ZFS. I use Apache everywhere, because it’s what I learned back in the 486 Age. My web site is critical to my business, so I must minimize downtime. I chose to implement a Caddy reverse proxy, because it looked easier than Envoy or migrating to nginx. (Nothing against either tool, of course.)

These are my notes, not a tutorial. If they help you, that’s grand. I pillaged Thomas Hurt’s post for this.

QUIC for HTTPS runs on UDP port 443. I suggest you start by opening UDP port 443 on your packet filter. Or you can follow my example, not open it, and spend half an hour staring at the screen shrieking why doesn’t this work? Up to you.

Caddy defaults to running as root, so it can bind to privileged ports. I played “run servers as root” in 1995 and have no desire to get rooted again, so I need to allow an unprivileged user to bind to privileged ports. That’s where FreeBSD’s mac_portacl comes in. It allows unprivileged users to bind to privileged ports according to a policy you set.

I’ve written about mac_portacl before, but my hosting architecture has changed. Instead of VMs scattered around the world, I now rent a single dedicated machine and use VNET jails. It saved me some money and gave me flexibility.

But mac_portacl is not jail-aware. You set rules per UID, but those rules apply across all jails. Individual jails can declare if they use mac_portacl or if they use the traditional scheme. You need to use consistent UIDs across all your jails, meaning that the user www must run whatever’s on port 80 and 443. All services need to run as different unprivileged users, so I’ll need to create a separate user for Apache.

Start on the host.

# pkg install portacl-rc

This gives /etc/rc.conf integration into mac_portacl. So go into /etc/rc.conf.

portacl_users="www"
portacl_user_www_tcp="http http"
portacl_user_www_udp="http"
portacl_enable=yes

Reboot the system and verify that portacl is working.

# sysctl security.mac.portacl
security.mac.portacl.rules: uid:80:tcp:443,uid:80:tcp:80,uid:80:udp:443,uid:80:udp:80
security.mac.portacl.port_high: 1023
security.mac.portacl.autoport_exempt: 1
security.mac.portacl.suser_exempt: 1
security.mac.portacl.enabled: 1

Looks good. Now go to the jail.

Control traditional privileged ports with the net.inet.ip.portrange.reservedhigh sysctl. By setting it to 0, you disable privileged ports.

# sysctl net.inet.ip.portrange.reservedhigh=0
net.inet.ip.portrange.reservedhigh: 1023 -> 0

Make the change permanent in sysctl.conf
net.inet.ip.portrange.reservedhigh=0

mac_portacl now controls access to ports 1 through 1023.

I need a separate user for Apache. Yes, I could run both as www but I survived the “run everything as nobody” era and learned my lesson.

# pw groupadd -n apache -g 81
# pw useradd -n apache -u 81 -g 81 -d /nonexistent -w no -s /usr/sbin/nologin

In theory, I can switch apache to run as this user and it’ll be fine. Reality will have a short sharp shock for me, I’m sure.

Now go to httpd.conf. Bind it to 127.0.0.1 port 8080

Listen 127.0.0.1:8080
User apache
Group apache

I also commented out mod_ssl. Apache will provide everything unencrypted, but only on localhost.

Go into the virtual host config. All those VirtualHost *:443 entries? They need changing to VirtualHost 127.0.0.1:8080

I also comment out all of the TLS entries. We’ll have Caddy manage TLS for us.

Restart Apache. Watch the error logs. It’ll gripe about a few files being owned by www. Change their owner.

Now configure caddy in /usr/local/etc/Caddyfile.

Permit me to put on my old sysadmin hat and shriek: “Don’t start config files with capital letters, people! You know better! WHYYY.” Yes, I know the world has moved on. Come closer so I can smack you with my cane.

Caddy does nothing except get X.509 certificates and forward connections to Apache.

test.mwl.io {
  reverse_proxy localhost:8080

  # Enable logging:
  log {
    output file /var/log/caddy/access.log
    format json
  }
}

That’s it.

Start Caddy. Point the browser at the HTTP site and it gets redirected to the HTTPS site. All this work, and I have replicated what I started with!

So let’s turn on QUIC and HTTP/3.

Sites must inform clients that QUIC is available through an HTTP header. The client makes an initial connection of HTTP/2, sees the header, and switches to HTTP/3 and QUIC. Add the header in the virtual host configuration.

Header set alt-svc "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600

Reload Apache. Set up your packet sniffer to watch UDP port 443. Point your browser at the web page.

So far, I like Caddy. It seems simpler than Apache. It is owned and backed by a commercial firm (ZeroSSL). I am careful going all-in on commercially-backed tools because the Internet’s business model is betrayal. I have other options if that happens.

I’ll deploy this on my main site to get some QUIC experience for the new Networking for System Administrators. QUIC isn’t essential today, but I want to future-proof it. I would be remiss if I didn’t mention that the book is open for sponsorships, for a little while longer at least.

Future path: do I need Apache? For some stuff, probably. But can I serve simple sites straight out of Caddy? Yes.

87: Farawayistan is Fubar

(Trouble this morning. Had to pull and republish this episode. Sorry if you see it twice, or if it didn’t work the first time.)

I really do need to get on the orc Christmas story. So here’s a bit from the networking book.

Enterprise environments have monitoring systems. Monitoring systems alert people. Those people recognize common alerts. “The link to Farawayistan is fubar again, ignore it.” Certain alerts are expected. “The ERP team started their patches two minutes early? The line manager will be annoyed—but that’s not my problem.” They only stir themselves for rare or unfamiliar alerts.

But some alerts trigger full-on alarm. “Port scanning on the secure subnet? My week is ruined, if I’m lucky!” When a second disturbing alarm arrives moments later, the sick feeling escalates to full-on panic. The reflexive coping strategy for panic is to share it. Eventually, an angry network admin or a furious manager or an icy-cold C-level shows up at your desk demanding to know exactly what you thought you were doing and if you can give any reason why you shouldn’t be fired. Even if you keep your job the network team will identify you as a problem—and they’ll be correct. You caused problems.

Remember, the word “fired” has multiple meanings. One is “to have your employment terminated.” The second is “to be set on fire.” Either might apply.

Or you can sponsor the networking book and remain unkindled.

Posted on

“Networking for System Administrators, 2nd Edition” Update

“The book is underway.” What does that mean? The first half is largely done. The Windows stuff is PowerShell. The Debian stuff uses ip. FreeBSD is the reference Unix. I wrote a chapter on what folks need to know about TLS, to go in the middle. I sadly sent that to Bob Beck for his comments. I understand where TLS has been, but Bob has good insight into where it will be.

Bob’s a good guy, but he also has cause for a lawful quarrel with me. Whenever I start to feel depressed, I listen to that recording of the BSDCan 2019 closing ceremony. That man screaming “LUCAS” in tones of rage and despair? That’s Bob.

He can now declare himself avenged.

If I’m talking modern TLS and modern networks, I really need to address QUIC. Not long ago, QUIC was largely theoretical for sysadmins. None of our common applications supported it. Yes, half of the traffic to Google went over QUIC, but I couldn’t deploy it. It merited a paragraph or two, no more.

Turns out that nginx main release now supports quic. I can reasonably deploy it. Except I’m using Apache. I’ve been running web servers since well before nginx, and never saw a reason to change. Yes, yes, Apache has issues, but I’m accustomed to its odd little habits and even the thing with the whirling razor blades followed by vinegar mist.

Could I skate by with three or four paragraphs on QUIC? Many tech authors do. I have this career because I don’t skate through, however. So I get to migrate my systems to nginx. Or perhaps stick an Envoy proxy in front of the web server. Haven’t decided yet. I’ll probably talk through the decision on the fediverse.

But it’s all Bob’s fault.

On the plus side, I have a sketch of the n4sa2e challenge coin. It’s not final, but it’s the way I’m leaning now.

n4sa2e challenge coin sketch

For the two people who care: this coin’s featured rodent is Willoughby. Lara Jean did a great job of capturing his character.

The only way you’ll get one of these coins is by becoming a print sponsor. Your backing will sustain me as I struggle with envoy. Or nginx. Probably both, then one.

I think the rest of the book will go quickly. You still shouldn’t nmap addresses you don’t control. DNS hasn’t changed, nor traceroute, nor netcat. I have to weave some TLS throughout, of course, which makes the netcat part a pain. (Standardize Netcat Flags and Give It TLS Challenge 2025!)

But at least Bob will have the pleasure of hearing me scream “BECK!” in blended rage and despair.

86: Take Quite a Long Time to Fail

Here’s a chunk of the networking book.

Standard netstat attempts to show hostnames instead of IP addresses. This means your server performs a reverse DNS lookup on every IP address it exchanges traffic with. On a busy server, this might mean hundreds or thousands of lookups. The output pauses for each lookup. Many hosts have no reverse DNS, so these lookups can take quite a long time before they fail.

Service names also appear with a human-friendly name rather than a port number whenever possible. It gathers this information from the services file. This results in a mix of named ports and numbers in netstat output, depending on whether a specific port has an entry in the services file.

All versions of netstat let you disable DNS lookups and port name lookups with the –n flag. I almost always recommend using –n. (I can’t think of any exceptions, but I’m sure there is one. Somewhere.)

Networking for System Administrators is open for sponsorships.

Posted on

More Titles in Direct Print Sales

In spare minutes, I’ve been expanding my direct print sale operation. You can now get all of these in my bookstore. If you pay for the print book, you get the ebook free.

titles available in print on tiltedwindmillpress.com, 21 April 2025

I have other books in the system, but am waiting for the print proofs to arrive. They come from a new printer (BookVault). Before I tell you to buy a book, I need to know that BV can produce the book as intended. They’re competent, but everyone handles PDFs slightly different. I’ve caught a couple weird color things and a skewed margin. So, despite my efforts to trim down in-house stock, I’m accumulating books. Dammit.

The thing I’m super excited about? Bundles.

It’s about eight years too late, but I now sell the FreeBSD Storage bundle in print. If you buy it from me, I can afford to knock 20% off. Even with shipping, that makes it a better deal for you than buying from Amazon.

My hope is that the kind of people who want to, for example, run their own mail servers will also want to buy directly from the author. That would help make up for the current, unforced and wholly unnecessary, economic implosion in the US.

Next up? The rest of the tech books. Discounted Cross-Platform Unix Mastery and Total Mastery bundles. Then all the fiction and finally, The Full Michael in print.

Updates will follow as more titles appear.

Posted on

“Laserblasted” Kickstarter over

It funded. My gratitude to everyone who backed, spread the word, or called me mad.

My goal on book Kickstarters is deliberately set below actual production cost. I want it to fund. I’m going to publish it anyway, and I’d rather get $500 to production cost than set a goal of the actual price and fail to fund.

I’d like to think that the US government deliberately decided to trash my campaign, but no. They trashed everyone equally. I’ve run enough Kickstarters that I know how they go. Kickstarter provides a graph of every campaign’s funding status. They all have very similar graphs. The dollar figures on the Y axis vary by book, but the shape is similar. Here’s my last campaign, Apocalypse Moi.

Every campaign funding has this shape. There’s an initial surge, a steady upward slope, and a final surge. Here’s Laserblasted.

That three-day dead spot in the middle is where the tariffs were announced. After that initial shock I did attract more backers, but other backers canceled their pledges or switched from hardcovers to ebooks. Again, I don’t blame them. But without that economic shock, the graph would have looked very different.

The good news? In absolute dollars, Laserblasted raised more than Apocalypse Moi. That’s cool. The bad news is that Laserblasted is wholly original, not a collection, and so expenses are much higher.

Laserblasted will be the first new release offered in print and ebook exclusively through my web store for a few weeks. It will trickle out to other stores.

Again, I don’t blame folks for not backing. When the plane loses pressure, put on your own air mask before helping others. This post is simply to tell others that they are not alone.