2.5 Admins 291: UPS for LiFePO4

This late-night Linux family podcast is made possible by our patrons.

Go to latenightlinux.com slash support for details of how you can join them.

Support us on Patreon for access to ad-free episodes and early releases.

That's latenightlinux.com slash support.

Two-and-a-half admins episode 291.

I'm Joe.

I'm Jim.

And I'm Alan.

And here we are again.

And before we get started, your custom reclire article plug Alan is

designing Open ZFS storage for independence.

Yeah, so in this article, we talk about some strategies you can

deploy to help build systems that aren't locked in and to make sure that you'll

be able to move your data around however you want, both from your existing system

to the ZFS system and then to your next system, whether that's ZFS or something else.

Open source and OpenStand, I'd say.

Basically.

Right, well.

And can the showroom says usual?

Please, please, please stop using past keys for encrypting user data.

Right.

Tim Kapali.

The major issue here is that if you have not taken precautions to make sure that you can

get back into any past key secure accounts and you lose or accidentally destroy the device

that is your past key, you get locked out all those accounts.

And a lot of folks don't really realize that aren't really planning ahead.

And if they just set up a past key because they're told a past key is what you should

do.

And then it seems convenient to do this and this and this and this and this and the other.

All with that past key, you might have the same situation I've recently complained about

with multifactor authentication when end users are just, they're end users.

They're not IT admins.

They're not always going to get it right ahead of time.

Think, well, I need to make sure I've got scratch codes or, you know, fall back methods

to get in.

They're just going to do what they were told to do and was told were safe.

And then when they drop their phone in the lake on a weekend, they can't get in anything

anymore.

Yeah.

I think in particular, in this case, they're talking about encrypting a backup of this

stuff from the app using the past key and it's like, well, the main reason I would need

to restore this backup is if I lose the phone and if this past key is tied very specifically

to this phone, then maybe that's not the right combination of security factors.

I don't know how we get this message really across to the people who need to hear it.

But I think the simplest version of the message that really needs to get through to the end

users of the world is you must configure fallback authentication methods.

You must do that.

You can't just be like, well, I set up the MFA and I'm done.

Nope.

You need an answer for what happens when you no longer have access to that multifactor

device or that past key device because it's going to happen.

And it's not the kind of thing you prepare for after the catastrophe.

Yeah, and I think the main point of this article specifically is past keys are designed

to solve the authentication problem to prove that you're the right person.

We shouldn't be conflating that with using it to drive the encryption keys for maintaining

the confidentiality of data that shouldn't be tied to the same thing as the authentication.

Right.

Because maybe you have multiple different ways to authenticate, but you need one more

recoverable way to get to the encrypted data.

And so to Jim's point, they're saying in the article, we cannot and should not expect

the users to understand this.

The wider authentication industry needs to stop trying to push this one solution to do more

than it was meant to do.

Right.

It was meant to solve the problem of people having a lot of different passwords, but it isn't

meant to solve the key management problem of encryption.

Yeah.

And Tim is very clear in his piece that he thinks that past keys are a very good idea for

authentication.

He has nothing against that.

It's just this particular use case.

Yeah.

And that the recovery he's and so on that you need to make sure that you can restore your

encrypted backup when specifically the most likely thing you've lost is the device that

this passkey was tied to.

And if you're fair, Tim might have no problem with past keys, so as authentication devices,

I have some issues.

The quantifiable value add of using a passkey rather than using, you know, simple MFA with

a login is that unlike two factor authentication in which a scammer might be able to convince

a victim to give them the one time code that then the attacker can use or convince the

attacker to just go ahead and, you know, tap the button, you know, with the right numbers

on it on their phone, you might be able to confuse granddad or grandmother to do that

if you're a nerd to well.

But if instead, grandma or granddad has a passkey, then a remote attacker can't convince them

to unlock the session that remote attacker is operating because the passkey is not in

the physical location.

It would need to be to do that.

That is a genuine security ad, but I would like to point out that's really kind of only

a genuine security ad for terminally clueless people.

And again, you know, if this is a security solution that is specifically most useful

for terminally clueless people, it really needs to be designed around the idea that that's

who's going to be using it and we need to make sure they can use it safely.

Yeah.

And not overload it and try to make it mean other things that it wasn't meant to do and

cause those same users more grief.

Isn't it wild for how many generations we've had computers now as a species?

And yet we still haven't figured out how to build an actual large scale computer system

that's usable by the humans who we intend to actually use it.

We haven't figured out where on that dial things should go.

The dial that is on one end security and on the other convenience.

Well, that's the thing.

There is no one place it goes to hijack your analogy.

It's a knob because you need to change the position on it pretty frequently.

If there was only one position, it wouldn't be a freaking knob.

You'd have just soldered a resistor into the board and called it a day.

Well, I suppose what I'm trying to say is we haven't figured out how to figure that

out for each application.

That is certainly arguable.

I'm feeling a little cranky today after dealing with a lot of different things and might

take tends to more be that, you know, we just don't care enough to bother and we never

have and we still don't and I'm not sure when we're going to start.

I mean, none of this stuff is anything that can't be figured out ahead of time.

We're not sitting here doing like pages of advanced like tensor math to come to the conclusion

that something was designed wrong.

It just like, we're sitting here and pointing out obviously freaking dumb things.

So why did we do the obviously dumb thing in the first place?

The only answer that makes any sense to me is, well, we didn't care about it being

dumb.

We didn't have enough to schlep out the door, so we did.

About a year ago, we talked about anchor, solex, portable power stations and how you can

use them as a UPS.

We've all got them and now Alan, you've bought yourself a new one, a C1000 Gen 2.

We're not sponsored.

You've used your own money for this.

You're pretty happy with it.

I understand.

Yeah.

So when we were talking about it a while ago, you know, Jim and I have been complaining about

the price of new UPSs or specifically batteries.

You know, I have a bunch of APC backups, 1500 Gs and, you know, it's been to two and a half

years.

The batteries are all dying and it's like, why am I spending more money on lead acid batteries

that are just only at the last two years?

So about a year ago, when we talked about it, I tried out blue tea.

I think it was 400 watt hour or so.

One of these portable power stations, but it turns out with that one, if you're pushing

more than about 60 watts through it, a little fan turns on and makes a lot of noise,

which would be really poor for podcasting.

So that had been relegated to powering the switch and the modem and so on in the laundry

room kind of my utility closet.

And I hadn't really found a very satisfying solution.

And then I saw an ad for this C1000 Gen 2 that specifically mentioned they had cut the

response time down.

So with the older generation of these power stations, the UPS cut over time was 20 milliseconds,

which seems good enough and I think almost any computer will be fine with it.

But it's a lot higher than the response time on a regular UPS.

But with the new Gen 2's, the response time is cut down to 10 milliseconds, which is a

lot more interesting.

It's still, I think a UPS is normally around 5 milliseconds, but it's a lot closer.

And you know, if you're talking about 60 hertz, it's less than a whole cycle.

But it's not up to 20 or up to 10.

Yeah, I think it's up to 10 or you know, I'm most 10 milliseconds for the cut over.

Yeah.

And mostly it was on sale on Amazon and it looked interesting and I was really, you know,

my UPS was beefing because the battery was dead.

So I'm like, I'm going to give this a try.

So I got it and I installed it and set it up for the router, which is actually a whole

like 2U server in my house.

Of course.

The two and a half gigabit, slash 10 gigabit, PUE switch that powers via PUE, all the access

points at my house and fans out to a bunch of stuff.

And the 10 gig switch that connects all the servers together.

So it adds up to like 250 watts current total draw all the time.

And one of the very nice features that the anchor app has for it is peak shaving.

So where I live, we have time of use power billing where I pay three different rates depending

on the time of day for power.

So the battery can actually say, all right, it's on peak time now.

I'm going to use the battery instead of the wall to get power to power my computers up

to the point where the battery hits, you know, 20 or 25% or whatever I can figure it to.

So that was me to kind of shift the constant load of my server to a different time of the

day when power is cheaper, which is just a nice add on.

And of course, you could do something similar with solar panels and stuff.

Yes.

Solar input where you could actually, you know, charge the battery not off the grid during

the day and so on.

But just the peak shaving, you know, I'm only going to save like 20 cents a day.

So it's not going to add up to pay off the power bank anytime soon.

But it's every little bit on top of my UPS wasn't ever going to pay for itself.

So, you know, this seemed pretty interesting.

And then I actually experienced a longer term power outage.

One of the reasons why I hadn't bit the bullet on getting something really big in a big battery

system for a whole house or something is, you know, in the 15 years I've lived in this

neighborhood, there's been one prolonged power outage and it was caused by like a car

physically tackling the telephone pole at the corner and knocking the power out for like

12 hours.

I was upset that the person who crashed into the telephone pole was out of the hospital

before the lights were back on.

In this case, it was just a fault with the transformer or something.

The power went out at 9.20 pm and didn't come back on for a couple of hours.

Originally, they estimated it wouldn't come back until three in the morning.

So I was glad when it came on a bit early.

But this power thing suddenly became a lot more critical where I was like, if the power

doesn't come back on soon, I'm powering off the router and the internet and moving the

portable power station to power my refrigerator instead.

Yeah, that's my plan.

I've got the C300, not the C300 DC, which doesn't do the UPS stuff.

That's only for kind of charging laptops and stuff.

But yeah, the C300, I've got it on my backup box and my access points and everything.

But if the shit hits the fan, then it's going to potentially power my heating system,

my fridge and the key stuff that I need.

I went in slightly different direction than Alan when I bought my anchor stuff initially.

I bought a handful of C300s and one C800 to go in the rack downstairs.

The C800 powers my big workstation, my server, a power of ethernet switch, a couple monitors,

like that.

The C300, which is the first one that I tested and tested very extensively.

It's only got a 288 watt hour capacity.

I say only, but that's sufficient to give a, it was a 9th generation i7 workstation, a

24-inch monitor running an artificial workload pulling 100 megs a second random off a solid

state drive.

It kept that up and running for three and a half hours.

An APC 1500 volt amp UPS would be lucky to keep that set up running for a half an hour

when it was brand new.

After it's a year old, I guarantee you it's not going to have a half an hour of runtime

on that.

Now we look at the newer stuff, the C1000 Gen 2.

You can pick that up for about $430, where at the time I was spending about $250 for

my C300s, the C1000 Gen 2 has a 1024 watt hour capacity on it.

That's a lot.

Anyway, the thing I want to get out with my slightly different strategy, I do have a

lot of really ugly power outages where I live in the American Southeast.

We have all aerial power lines.

We have tons and tons of trees with very iffy arbor maintenance keeping tree branches away

from lines.

So if there's a lot of wind or a heavy storm, it is not uncommon to have power outages.

And when a hurricane rolls through, well, last year my family was out of power from

more than a week.

Luckily, I already have this giant grab bag of high capacity power banks with AC capacity.

And just down the road at the assisted living home that my parents live in now, they did

have power.

And that meant that anytime we wanted to, we could grab a handful of very easily portable

devices with really easy grab handles, their own trouble lights, everything.

We could just go visit Mom and Scott, plug them in for an hour or so over there, charging

back up, bring them back home, run whatever we wanted to off of them.

It makes a really, really big difference, not only for, you know, the brief power outages

that you normally think of having UPS for, but if you've got enough of these things and

enough capacity, it can be a real big disaster survivability piece equipment.

Yeah, try doing that with an old lead acid UPS.

And also the lead acid UPS, you're not going to have USB ports on it to directly charge,

you know, phones, tablets and whatnot.

Instead you would have to plug your AC charger adapter into the UPS to then convert back

to DC to charge your devices, which is just brutally inefficient and slashes, you know,

that runtime slash capacity even further.

Whereas here, you've got direct DC charging for devices that want DC.

You've got nice AC out for devices that want that.

And you've got a really nice trouble light with multiple brightnesses, you know, for

when again, you know, the power is actually out.

Turns out that's pretty frequently a really nice feature to have baked in.

It's amazing.

You mentioned that the one thing you lose with the Gen 2 of the C1000 is it doesn't have

a light anymore, but it is like 11% smaller and quite a bit lighter.

It comes in at like 24 point something pounds.

I was almost surprised how small it was considering I was getting, you know, a kilowatt

of a kilowatt hour of power in it.

Of course, having had the power outage and kind of wished to have more power and especially

watching it peak shave each day and noticing that when it also powers my media server as

well, the battery doesn't last long enough to survive the whole peak time.

And so I'm still paying for a bit more of the peak time power.

I started looking at the bigger one and they have the C2000 Gen 2.

I think it's just around $800 US, so not double the price for double the capacity.

And that looked interesting.

Then I started looking at previously we would talk about their like F 3800 kind of whole

home but portable version.

And I know at least one other listener has written in about their experience.

I actually bought a pair of those and wired them up to the house and they were pretty happy

with them.

But I also noticed that anchor announced their new like E 10 system that's like a testless

power wall type mechanism, something big that you install in your garage to power the

whole home.

And so I'm still looking at options.

The other interesting thing is the integration with home assistant.

I can actually pull all the stats out of anchor into home assistant and like I can actually

see graph of the battery charge and see the peak shaving happening and all that integration

and even configure it.

So I can have automations is that like, Hey, a storm is coming.

Don't peak shave today, save all the battery in case the power goes out and things like

that.

How are you connecting it to home assistant then?

They have an integration that's not great, but there's an MQTT based one that works

very well.

So I just got the anchor solics thing out of the community store and it works very nicely.

Okay, this episode is sponsored by Automox.

If you're evaluating endpoint management solutions, the real risk isn't which tool you choose.

It's choosing one without a proven blueprint for success.

Because while most endpoint tools promise automation, next to none promise outcomes.

That's why Automox introduced turnkey results.

Turnkey removes the uncertainty that holds endpoint automation back.

It delivers a personalized results blueprint, so Automox is configured and operated to

achieve real outcomes from day one.

No trial and error, just clarity.

Each blueprint is validated across millions of endpoints managed by Automox and tailored

to your specific environment, your risk tolerance and your operational goals.

Instead of guessing how aggressive to be with automation or how to configure policies

safely, start with the plan you can trust and you can choose to implement the plan yourself

or have Automox do it for you.

Either way, you get faster time to value, lower risk and predictable endpoint outcomes.

Automox turnkey results.

Know the plan, trust the result.

Learn more at www.automox.com.

Let's do some feedback.

Christian writes, I know nobody on this show would like to put their mission critical

data on butter FS, but what about using it for the root file system?

That way you can still take snapshots on the root and you can still use that FS for everything

else, right?

As long as it's a single disk file system and you're not using butter raid and I would recommend

avoiding compression and don't rely on replicating it anywhere, but with all that said, if that's

still something you want and you're looking for my personal blessing, you have it.

Be free.

Or on your root, take the occasional snapshot.

Be happy.

It just seems like you're going through a lot of contortions to try to get a little butter

in your file system when it's probably fewer contortions to do set of this boot menu.

I don't know about that.

There's a lot of distros that default to butter FS on the root like Fedora, so is a, you

know, ZFS boot menu is really not an end user easy level solution.

Is it reasonably simple for an experienced, you know, IT person to set up?

Absolutely.

My entire family are full-time Linux users and I would not tell any single one of them,

hey, I want you to go install a machine with, you know, ZFS boot menu for me.

No, that's, that's not going to work out.

Yeah, but they could probably work out the Fedora installer, which would just, by default,

give them butter FS.

Any of them would be able to do that, yes.

Before I ever met my wife, she had actually had a friend and roommate have a Windows

installation go completely bonkers.

I think it got malware to hell and back.

I don't know.

This was long before the two of us met and she went on the internet and discovered a

Ubuntu and downloaded it and installed it on her friends laptop and her friend became

an Ubuntu user.

Janice is not an IT person.

She's just, you know, a competent, reasonably self-confident person who saw a problem

and resolved it.

So, yeah, that's totally doable.

I don't want to dog on ZFS boot menu.

I think it's a really compelling product and I think within the limitations, they're

forced to operate under.

They've made it reasonably easy to use and you wind up with something really awesome

after you go through those steps.

But calling it easy, nah, I got to kick up my feet on that one.

Well, yeah, I guess my point was so much it was easy.

It was like, if you want the features, maybe it's worth a little bit of extra pain, then

getting the half-arrest implementation you would get from butter FS.

It's like, at what point do you just stick with the default, like EXT file system?

I guess it's a bit of a different story if you're distro defaults to butter FS and

you'd have to opt out of it.

But if you're opting into butter, it's like how much more effort would it be to opt into

ZFS instead?

A fair amount.

Because you have to go through a debut strap, you know, command line installation.

You're not using like the normal installer at all.

You got to, you got to munk your end with charutes and debut strap and you wind up with

like a weird system that doesn't quite look like a typical desktop or server from any

of the builds available from the normal install.

And like, can you, can you work with all that?

If you know what you're doing, yeah, you, you can.

But I'm not even calling that simple and easy for me, doable, absolutely, simple and

easy.

Oh, hell no.

I would argue that it's, honestly, it would probably be easier right now if what you really

want is you want an easy way to go ahead and get ZFS on root with boot environments.

Make switched free BSD.

Let's do some free consulting then.

For first, just a quick thank you to everyone who supports us with PayPal and Patreon.

We really do appreciate that.

If you want to join those people, you can go to 2.5abbins.com slash support.

And remember that for various amounts on Patreon, you can get an advert free RSS feed of either

just this show or all the shows in the late night Linux family.

And if you want to send any questions for Jim and Alan or your feedback, you can email

show at 2.5abbins.com.

Quentin writes, I've been wondering about restricting internet access for some of the IOT devices

on my network that don't normally need it.

As a home automation enthusiast, I have a lot of them.

Obviously, a VLAN is the standard way to do this, but I've been wondering about a tactic

that would be easier to implement.

Suppose my DHCP server was configured not to give out a usable gateway address to the

MAC addresses I'd like to restrict.

This is just security by obscurity, and a dedicated attacker could bypass it easily

by searching for routes to the outside world, but I doubt that your average IOT device

wanting to call home to a Chinese server would do such a thing.

Nice, my brilliant solution becomes more widely adopted, of course.

You don't really need VLANs for this, but I don't really approve of the idea of just

handing out a bogus gateway address.

As Quentin mentioned himself, that's very easily bypassed security through obscurity.

A better and arguably just as easy way to manage this, maybe even easier and more discoverable

later, is you just set up static DHCP leases that always hand out the same IP address

to the devices that you want to restrict.

When you set up firewall rules, not to allow those devices to talk to the internet, you

don't need a VLAN.

Yeah, so what you could do here is, as Jim said, create static entries for these servers

in your DHCP configuration, or not servers, IOT devices.

You get the bonus of all your IOT devices having static IP addresses, which is handy for

your home assistant to manage them and so on, but if you were to do that in a certain range

of IP addresses in your subnet, or as a whole separate subnet, but that gets complicated.

If we just say, you know, everything from 120 to 150 in your subnet is IOT, then you

just have a firewall rule that says anything in that address is blocked and can't go out

to the internet.

And your DHCP server only gives out that range to those MAC addresses and you're good

to go.

It's just change the range in your leases for regular devices to not include that set

of IPs and statically assign them to those devices and block it and it'll be a lot better.

The only reason that you need the VLANs or otherwise implement multiple subnets, which

is really all VLANs are, they're a way of implementing multiple subnets without needing

an actual router in between.

The only reason to do that is not to restrict devices from getting to the internet, to restrict

two different devices on the same local network, arguably, from being able to talk to one

another.

Because one way or another, when you leave the subnet is when you have to go through

the router and therefore the router gets a chance to implement a firewall rule.

You can't just block off one range of IPs from talking to another range of IPs on the

same network because they are allowed to just speak to each other directly and that's

the way they're going to do it.

They don't send their traffic through the router in the first place.

But again, if all you're worried about is saying you don't get to go to the internet, that's

what firewall rules are for.

No extra subnet required.

Yeah.

And I would caution against the DHCP lease that purposely breaks things.

If you give out a bogus IP, you're going to keep seeing ARP traffic where all the IoT

devices are being like, what MAC address is the router and there's nobody there answering

so they're just going to be spamming the network all the time.

And if you're on your 2.4GHz Wi-Fi, there's going to start eating up more and more airtime

and making that side of your Wi-Fi even worse.

But also, some of the devices have a failsafe where it's like, hey, they'll either try to

guess and try to find the internet another way or they'll be like, this isn't working,

I'm going to reset and try to find a connection again.

And that might mean that they fall back into a safe mode or other things that just caused

the IoT devices not to work the way they're supposed to.

So I think Jim's suggestion is a lot better.

And would have saved me some hassle.

I was doing it with VLANs and I forgot to change the tagging on one of the VLAN ports

and my IoT circuit transformer clamps that go on in my circuit panel to measure the power

of all the different circuits.

I had fixed the config but didn't copy run start on my switch.

And so when I had a power outage, my switch went back to the old config where that ethernet

port wasn't in the right VLAN and suddenly I had no monitoring of my electric panel and

I had no idea why.

The other thing that we should warn you about kind of long the same lines as Alan talking

about devices kind of freaking out and trying things to get the internet is these days an

awful lot of things even by default right out of the factory.

They'll spoof a Mac.

They'll give you a random spoofed Mac every time.

And you can't use any of these tactics with a device that isn't always reporting to

you with the same Mac address.

In those cases, if you've got a device that randomizes the Mac address and you don't

have a way to disable that or you don't trust the control that says it's letting you

disable that, then you're back to like, you really do need entirely separate networks

whether it's a VLAN or via, you know, literally separate physical segments, you have to break

them out because you're reliant on that Mac address being unique and not changing to

allow you to identify the device on the network and, you know, thereby do different things

to it than you do to the other devices.

Yeah, depending on your situation, it might be easier to enumerate and give a static IP

address to all the machines that should have the internet and leave the default everything

that's not in this blessed range of the subnet, not having the internet, but that might

guest Wi-Fi really annoying at your house.

This is why I have a separate SSID with a separate subnet on a separate VLAN for all my

IoT stuff because I want to keep it all separate and that SSID is 2.4 gigahertz only and

my phone knows it's not to ever try to connect to it.

I know for some listeners, the idea of setting up VLANs seems relatively trivial.

For some listeners, they already have Wi-Fi gear that supports, you know, arbitrary numbers

of VLANs and they can spin up whatever they want and they're comfortable with it in those

folks.

Sweet.

You got all the tools you need.

You got the knowledge you need.

You're ready to go.

I also know we have a lot of listeners that are like, oh my goodness, that is a lot.

I barely know what a VLAN is.

My gear doesn't support that.

Ah, I don't want to learn all that.

Well, there is a much easier answer, which is just by a completely separate Wi-Fi access

point on a different SSID and now you can control things because everything is coming onto

your real network from that other Wi-Fi network.

So you've essentially created a guest network just by buying, you know, a different AP

and putting it with a different SSID and password and that's what you put your IoT devices

onto and they have no way to jump off of that and get somewhere else where they won't

be tightly controlled.

Yeah.

And that's a great use for your old AP when you've upgraded the new ones to have six

gigahertz and so on.

Maybe keep the old one around and use it just for IoT crap.

Most of it only supports 2.4 anyway.

Yeah.

Works in our industry like to laugh at Stonac solutions?

Not me.

Stonac's work.

We use those things for a few thousand years for a reason.

Yeah.

And I'm definitely done with Jim here, the idea of doing this with just a firewall rule

is a lot nicer, especially when you have switches that aren't managed and you can't

easily do a bunch of VLANs and get overcomplicated.

Even if you wanted to.

It's like I spent all that money on smarter switches when I can just write a firewall

rule and I have to bother upgrading all my infrastructure.

Right.

Well, we better get out of here then.

Remember show at 2.5 admins.com.

If you want to send any questions or feedback, you can find me at joarest.com slash master

dawn.

You can find me at mercenariesacademy.com and I'm at Alan Jude.

We'll see you next week.