A random day in February

Nice day. Made spaghetti – we make enough for more than a few meals later, so it’s a half-day effort. The whole upstairs smells delicious, but I couldn’t eat another bite.

Ran a new network line to Christina’s new office – this required 2 panels cut into the drywall, and one hole drilled in the floor. Turned out really well, and not a chaotic mess. Now only the kitchen and living room lack network cables, but our wifi mesh is pretty good, so that’s not a today kind of task.

Also finally decided on a larger synology NAS to replace my storage server. Moving from a server to a NAS seems counter intuitive sometimes, but other than a few apps that will run as docker containers, it just needs to store things, and NAS devices to that well. Plus they tend to be quiet, and the hotswap bays are nice.

It IS going to mean some future server rack re-org, likely removing EVERYTHING from the rack, then adding once at a time — an early summer project perhaps, ’cause that’s a whole day of downtime, guaranteed.

Alan Watts

I stumbled across the text for this a little while ago, and I know it will be a joy to come back to and read again later.

To be a human being you have to love the light, but you also must trust yourself to the darkness. Be able to let yourself go in the faith that you’ll arrive back all in one piece.

Now, it’s alright. The ground is going to hold you up. So just just lie down. And there’s nothing else you need to do, because the ground is firm and it will hold you there.”

“Look, do you know what you’re doing? You’re trying to hold yourself together. As if your skin weren’t strong enough to contain you. And you’re doing this all the time to keep yourself from falling apart. Why? You think, do you, that if you don’t hold yourself together you’re just going to go bleah and disappear into some kind of frightful green jello? No, you won’t.

Life update

So I was going to post this on facebook.. but then no. I barely use facebook these days, and there’s little value to dropping any more data into that sinkhole.

I might’ve posted it on reddit, but it’s a slice of life kind of post, and I don’t really want to expose that for cross referencing, I already give enough life details by accident.

I might’ve posted on one of the new sites, post.news, etc, but they’re really new to me, and I don’t know that I want to start doing that somewhere I may not stay.

So, here I am.

I have this week off. It’s the first time in a while I’ve had this many days off, and it’s been fantastic.  I’ve continued my efforts to replace all of the initial network wiring I did .. back in 2014… when we got the house.  Yeah, I know, keeping hack jobs is bad karma, but it’s been working, so it was hard to justify changing it.

I have maybe 2 or 3 more network cables to replace, then I’m done with that kind of wizardry for the near term.

I have a patch panel coming for the main rack.. getting it in place will require more than a few changes to the rack layout, I’m not even starting that until mid/late January however.

I’ve discovered the bathroom counters we JUST had replaced.. suck.  MDF anywhere near water is a bad idea.  I had a bad feeling, but I got sucked into that sunk costs fallacy, and didn’t cancel that part of the work before it went in.

Now, we pay for it twice I guess.  Second time will be with Floform, who did our kitchen counter (and wow, amazing).

Also need to replace the garage door this year – I’ve put it off and put it off, but I really shouldn’t wait until I don’t have a choice.  I have a few companies in mind to call this spring to get quotes from, we’ll be doing one of them for sure by summer.

We also need to again look at a new washer/dryer — these ones are occasionally frustrating, and the dryer is starting to sound like it’s wearing out.  A not surprising expense, they only seem to last a handful of years, and we use them every day, sometimes several times a day.

I have additionally considered getting permanent A/C for our bedroom — I think that’s going to slip a year, but I do want to make it happen. Portable units .. work, but they are bulky and somewhat loud.

Similarly, I have a plan to get our windows replaced – they’re not outside of their lifespan yet, but some are in less than great shape, and better to do them while we can still afford it.

Work is .. well, working out.  Some future plans are slowly coming together – winning the lottery would help, but in the meantime the slow path is still progress.

Puppet to saltstack

With Perforce acquiring Puppet, I finally found the motivation required to really look at my configuration management system and imagine replacing it.

Saltstack came up as an option. I almost immediately started to appreciate how easily you could schedule a run, run on a minion, or trigger a minion run from a master. Being able to target specific commands or sls files is amazing.

I was a bit shocked at how few manuals are out there, and how most of them were written ~2014. The prebuilt formulas are also a bit deprecated, and in some cases they’re just completely busted.

Still, the tools work. The docs at saltstack are good – not perfect (some items are mentioned briefly but not detailed), but still good enough to serve.

As of today, I have 51 formulas – about 30 of those are community formulas I forked to either a) make the code function at all b) add support for more recent versions and config settings or c) cover my specific edge cases.

At this point saltstack manages 17 hosts (including itself). It manages configs for powerdns, zabbix, telegraf, samba, nfs .. and my entire mail suite.

The biggest challenge I faced was inertia – this kept me from converting earlier as well. Salt applies configs for a minimum of 200 items per host. This is up to 400 on some very complicated hosts .. and all of those pieces required configuration, be it SLS files, pillar, or grains.

Similarly though, once I’d reached a certain point in this journey, inertia started working with me. I wanted to trial loki and promtail — it took about 30 minutes to write a formula for promtail to call into loki. It’ll take seconds to roll that out to my hosts.

It took about 3 months of casual tinkering to get the components duplicated out of puppet and running on salt. When I cut over, I simply removed and purged puppet, and ran the salt bootstrap. A few minor errors popped up, but by this point I knew how to fix them.

A note, I actually deviated a bit from the norm. I went with Pillarstack over pillar for most of my configuration. I found the yaml syntax did what I needed, and the very few places I needed SLS, I used pillar. It works.

I’m a real fan of how lists are processed *in order*, so my list of roles for a given host in pillarstack apply in that order, every time. Puppet would do them consistently for a given host, but not identically across hosts.

I’m a fan of how you can piggyback another value or value set onto an existing setting in pillarstack (for example add host specific path to a common list of paths for backups)

I’m definitely not using Salt to it’s limits, there are whole areas which it supports that I haven’t touched. I also don’t use (don’t currently need) separate salt environments, though it would be pretty handy if this managing a product, instead of my lab.

It was a lot of effort, but it was worth it.

Saltstack adventures

Still working through Saltstack. Definitely a small community at work on this, but pretty regular software updates. In some ways it reminds me of Ansible and how it can be challenging to find a bit of prebuilt boilerplate to meet a specific need — I’ve been a bit spoiled in that regard with Puppet.

That said, I’ve been making good progress. I have a minimum-spec list of things I need this to do, and every day I’m able to cross off another one. I’ll admit I’m quite drawn to a client-server model that’s *push* and *pull*, not just one or the other. I’m not sure why it’s not more popular, that combination alone makes it tremendously appealing to me.
It’ll still be some time before I try applying this to any of my working hosts, and some (like my mailserver) will be a massive undertaking.

It’s taken a bit to get semi-comfortable with Jinja templating, but it’s similar in many regards to ERB, so other than how ugly each looks prior to processing, I have no real complaints.

I did have to discover the difference between {{ var }}, {{ var | json }}, and {{ var | yaml }} — which for ordered lists comes out as [‘var’], [“var”], and [var] respectively. Yes, I put that here so I can find it again later 🙂

Next day, through the gauntlet.

We spent today busy. Rightly so, yesterday was a bad day. Today we painted the inside of the new shed. White. Christina had already given it a coat with some paint we had left over, but she didn’t have quite enough. This time we both went to town on it, and .. well, it’s white now!
She also painted the door (red). We have been cleaning up .. mementos as part of getting through that ‘punched in the gut’ feeling you get when you find an object that triggers a memory. Mostly that’s all cleaned up.
We’ve also both agreed that hell no, it’s too lonely just us. We’re going to see if we can’t adopt another pair of kittens. Turns out the rescue society we found Shamus and Bongo at still exists, and they’re still rescuing cats in need. We’ll go that route again.
That gives us something to look forward to, which is a heck of a lot better than sitting here looking backward and second guessing signs and clues, and regretting our losses.
That doesn’t mean regret doesn’t still happen, but at least it’s not the only thing.

My poor Bongo.

I’m glad this is a long weekend, I don’t expect I’ll be good for much for a few days
Tomorrow afternoon we bring Boo back to the vet. Final trip. I am witness to how accurate the vet was. He said weeks, but in just a few days Boo has started limping more, drooling, and yowling. These are new things, and they highlight how goddamn fast this is moving.
Weeks, maybe. Horrible weeks.
It’s been 13 years and 3 homes. Of the 7 animals we started with, he’s the last. Our final bright spark. So much bigger than the tiny kitten who hid behind the toilet shaking in fear when we brought him home.
I will miss him.

2020 is the worst year ever.

WordPress is being junky with this post.

It’s helping to replace my grief with rage.

Bongo has cancer.  We brought him in to get checked for rapid weight loss, but now we are looking at at most weeks before he dies of this shit.

We’re bringing him back on friday so they can euthanize him, which is a damn sight better than dying in agony, which is apparently the only other option.  He’s already been too sick for surgery, he probably wouldn’t make it through this, and if he did his immune system is suppressed (intentionally due to an autoimmune disorder), so the first infection he got would do him in.

I hate this.

I hate it.

My poor cat.

I have to write something, so that Rosey isn’t the first story I see.

I haven’t written in a while, and coming back to discover my grief when Rosey died is a quiet punch in the gut.

I know this post isn’t much better, I’m still referencing it, but what can you expect?  It’s still raw.  I still walk into the room and expect her to be there.  I make a noise and turn to see if it woke her.

I’ll get past it.  I don’t think I’ll get over it.  It’s almost Bongo’s time as well, he’s getting thinner pretty rapidly, and he doesn’t eat much at all.  All of the steroids in his system (that prevent the autoimmune disorder from eating him alive) have consequences that are inevitable.

I’m hoping we take in 2021 with him, but I’m not certain.  I find myself wondering “How bad will he need to be before we bring him in?”.

Leaving him to eventually expire is not an option.  I shouldn’t have left Rosey to that fate.  I won’t leave Bongo to fight for one more breath.  It was agonizing to watch, I can’t bear to subject our Boo to that as well.

That does of course lead to the thought that I’ll probably want to find a similar service for *me* when everything finally breaks down.  There’s no victory in trying to fight inevitability.

I still have a long time before I need to consider that.  If I’m lucky, maybe 40 years?  That’ll be enough time I think.

 

Roseybug has died.

Oh god, I have to write something here.  I’ll try to get through it, preferably without bursting into tears again.

This morning Rosey moved from her play space back into her cage, near her water.  She wasn’t very alert, and wasn’t always aware when I touched her.

She found a comfortable position braced against a corner of the cage, and spent most of the day dozing.  Around 2 though she was fully sprawled out.  She started seizing.  I hope she wasn’t conscious by this point.  I called Christina, and she came over and sat with Rosey.

I won’t lie, I couldn’t do it this time.  I had to walk away, and tears were streaming down my face.

It took about 30 minutes from first seizure until her last breath.  If there’s any mercy in the world at all, I pray she didn’t feel any of it.  It was horrifying though, her body struggling to last just one more moment.. and failing to.

It took some time for me to be able to speak without my voice breaking uncontrollably.  I finally managed to call the vet, and we took her in for cremation.  I couldn’t bear to leave her there untended for a single moment.

When we came back we spent time taking down her cage and removing her pen.  Nobody else will ever use those things now that she’s gone.  If we ever have another rabbit, they’ll get a new environment.

Now my space is uncomfortably empty.  The space she occupied … she’s been there since we moved here in 2014.  There’s never been a moment there wasn’t a rabbit in that space, and it breaks my heart that now the space stands vacant.

I’ve made it this far with just a few tears. I’m stopping while I can.