To add to Onomatopoeia’s excellent post, separate devices also limit the blast radius of any compromise. Attackers pivot when they compromise a system. They use one system to talk to others and attack them from inside your network. So you don’t want everything on the same OS kernel.

Unfortunately I don’t feel like I’m qualified to say what works well yet, not until I have the pieces of my site put together and working, and vetted by whatever security professionals I can get to look at it and tell me what I did wrong.

But right now I think that looks like every service VM on its own VLAN on a /30 net, and ideally the service VM and firewall/router VM serving it on different physical hardware joined by a managed switch. That managed switch shouldn’t let either VM host touch its management VLAN, and (I think, I don’t do this yet) should send monitor traffic to yet another physical host for analysis.

(“I can see why you’re not done yet” - yeah I know.)

Regarding the Lone SME thing, my wife has already told me if something happens to me, all my server stuff is getting donated. I should not expect her to maintain it after I’m gone. And I don’t. That’s entirely reasonable. If it lives on after I’m gone it’ll be because the recipe thing was useful enough for others to maintain. My specific server and domain kinda don’t matter.

This is my dream as well, but for security I feel like you need multiple independent systems. I’m doing mine with power-hungry recycled 2012-vintage server hardware (Xeon E5-1620s and 2620s and Opteron 6276s, bought for $100 each several years ago, plus a few hundred more to their maximum amounts of DDR3 ECC) but this hypothetical box could easily have raspberry pis or something similar. Public services can become compromised and you’ll only want certain hardware to be trusted to do certain things.

My plan is a terrible one and I’m taking way too long to do it. I really want someone else to build this better and faster, but if my crappy plan ends up being the first usable version of this, that will suck but at least it’s available.

I had a dumb personal domain from June of 2000, tried to make it a public internet site, offered services to people on IRC for internet social points, but after a few years it got ahead of me and I let it die. (I’ve been paying for the same business internet ever since, though, and I still have the same static IPs as from back then.) Time passed, got married, got a computer science degree and a development job with a billion dollar SAAS company.

I can see how they do big public internet hosting. I want everyone to be able to do this, too. Been trying to build the same kinds of architecture with open source tools at home. Struggling, I keep over designing it and getting stuck and frustrated. It takes me a month to do what a competent ops person from work does in a couple days.

OnceI have this working for me, I can share it, because it’s my own work product. It’ll be a guide, a recipe to follow, for creating the kind of secure and isolated web application and general VM hosting environment I see us use at work. This stuff is the difference between “I’m hosting one thing and if it gets hacked, everything is owned” and “I’m hosting a hundred things, all different, and if one gets hacked that will suck - but the other 99 things will stay safe.”

Biggest problem I think with creating this with open-source is just picking a direction for everything and getting the internet to not pitch a fit. “Why did you use postfix?” “I hate Greenbone / GSA and refuse to use it.” “Hardware is expensive, you say I need a jump box for this AND for this, and dedicated hardware for a firewall here AND here? Each of those could clearly be a VM. Your project wastes hardware and I’m not doing it this way.”

Sure, once this is done these decisions are pretty much baked in and I won’t have the energy to redo them yet again. But getting the architecture perfectly designed for your exact scenario … that takes a ton of work. Big companies pay a ton of money in just payroll hours to build this kind of thing bespoke for their needs. I’ll be giving away my version, and I’m afraid the internet won’t care.

But I think we need to keep this ability alive, that private citizens can set up their own DIY hosting that can stand up to hostile internet actors decently well. They can pay (I’ll grant) exploitative rates for business internet connections so they can have static IPs at home as well. If we all stop, we all just decide all hosting should be done by big cloud service companies or big enterprises, we lose a crucial bit of internet freedom. Someone needs to say “yeah this is kinda dumb but I’m doing it anyway.”

And if they could do it with a box you just plug in, instead of my (likely) month-long two hundred step recipe, and still have it stand up to attacks and “Internet background radiation” and stuff, that would be epic. I kind of don’t want my thing to be the way that self-hosting-public-web-services is done.

True. I kinda dodged that problem by having a personal .net domain that’s older than wikipedia.org. My understanding is that you can raise your domain’s reputation with some work.

Honestly the most important thing I use my domain for is easy-to-delete mailboxes and aliases to give to companies and contacts. That’s just incoming email.

For outgoing, there are services that let you send them an email and receive a report on any mistakes or misconfgurations they notice. I followed the first tutorial I found that didn’t seem like it was just advertising “see how hard email is? Looks impossible doesn’t it? Why not pay us instead.” Ended up being at linuxbabe dot com, run by Guoan Xiao, with part one titled “Build Your Own Email Server on Ubuntu: Basic Postfix Setup”. No links but search engines find it.

Big difference is I use OpenLDAP/slapd, and I put different components on different VMs. Took maybe a couple weeks of free time here and there, but I’m proud to say my outgoing emails seem to be accepted everywhere. Not that I send many, really.

Eventually planning on implementing filtering for terms and conditions updates for long-forgotten sign ups. I would like those to bounce.

I’d recommend looking again, as I think that advice is becoming dated. Greylist and DKIM make spam prevention super simple, ironically because the centralization of email towards Outlook and gmail has trained pretty much every sender to follow the rules or your email doesn’t go through. And then Greylist catches the rest, because spammers don’t come back and retry after a few minutes.

Devs make mistakes. We want to put up guardrails so mistakes don’t hurt us so much.

Please don’t deliberately line the guardrails with barbed wire.

N=16 developers

I feel like there should be a third box with Wall Street raider types, for scrapers that use Selenium browser automation.

I don’t think it’s entirely unblockable - adsense seems to know to only serve unmonetized PSA ads - but I think it’s very difficult to discriminate between “this is a real browser controlled by an end user” and “this is a real browser being controlled by automated test software”.

This.

My units and integration tests are for the things I thought of, and more importantly, don’t want to accidentally break in the future. I will be monumentally stupid a year from now and try to destroy something because I forgot it existed.

Testers get in there and play, be creative, be evil, and they discuss what they find. Is this a problem? Do we want to get out in front of it before the customer finds it? They aren’t the red team, they aren’t the enemy. We sharpen each other. And we need each other.

Love this, 100% accurate. QA people are amazing, protect us from ourselves in so many ways we didn’t even think of.

I feel like objecting to the “General advice about email is don’t” thing but I don’t know if I understand the objections well enough to refute them. I self host email for mspencer.net (meaning all requests including DNS are served from hardware in my living space) and I have literally zero spam and can’t remember the last time I had to intervene on my mail server.

On one hand: My emails are received without issue by major providers (outlook, gmail, etc) and I get nearly zero spam. (Two spam senders were using legitimate email services, I reported them, and got human-seeming replies from administrators saying they would take care of it.) And I get amusing pflogsumm (summarizes postfix logs) emails daily showing like 5 emails delivered, 45 rejected, with all of the things that were tried but didn’t work.

On the other: most of the spam prevention comes from greylist, making all new senders retry after a few minutes (because generally a legit MTA will retry while a spammer will not) and that delays most emails by a few minutes. And it was a bear to set up. I used a like 18 step walkthrough on linuxbabe dot com I think, but added some difficulty by storing some use and alias databases on OpenLDAP / slapd instead of in flat files.

But hey, unlimited mail aliases, and I’m thinking of configuring things so emails bounce if they seem to contain just a notification that terms and conditions are updated somewhere. I don’t know, cause some chaos I guess.

And I have no idea if my situation is persuasive for anyone because I don’t know what the general advice means. And I worry it’ll have the unfortunate side effect of making self hosting type nerds like me start forgetting how to run their own email, causing control of email to become more centralized. And I strongly dislike that.

I’m surprised I’m the first comment saying this, but all I see is a user who needs help expressing their needs but who is not getting that help. Sure they don’t have our experience with decomposing problems and anticipating technical issues, but that’s normal and expected.

Yep, mspencer dot net (what little of it is currently up, I suck at ops stuff) is 2012-vintage hardware, four boxes totaling 704 GB RAM, 8x10TB SAS disks, and a still-unused LTO-3 tape drive. I’ll upgrade further when I finally figure out how to make proper use of what I already have. Until then it’s all a fancy heated cat tree, more or less.