Selfhosted

63012 readers

2064 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

Local homeserver suddenly not responsive to SSH but can be pinged (piefed.ca)

submitted 1 month ago by gedfromgont@piefed.ca to c/selfhosted@lemmy.world

6 comments fedilink hide all child comments

This is something that seems to happen randomly to my little RaspberryPi 4B homeserver, like once every few months maybe.

What happens is that out of nowhere (meaning I didn't make any change to config or installation) I cannot reach any services I host on it (i.e. Jellyfin, Anchor notes...). I cannot access it via SSH at all, connection keeps timing out.

Ping however gives a signal. I can see/hear its running and doing something, since I hear the external HDD working. It is running headless and I have no monitor and/or keyboard I could connect.

So far my only way out is to just switch its power off, which is obviously not good for the hard drive. It has happened now like half a dozen times over the past few years, so I am wondering what a better way is to handle this. This was on RPI OS and now its happening on DietPi, so seems not related to the OS version.

I tried sending a shutdown signal via SSH but that also just times out, so how do I get it to shutdown properly?

Secondly, what kind of logs could I look at or start collecting to make sense of this?

you are viewing a single comment's thread
view the rest of the comments

[–] confusedpuppy@lemmy.dbzer0.com 1 points 1 month ago* (last edited 1 month ago) (1 children)

It sounds to me like ssdh may have stopped working. That may explain why you can't ssh into your server but pings still respond. I have a Raspberry Pi4 and a Pi5 and have had similar issues in the past.

I would probably approach this issue by writing a small script that checks every so often if the process sshd is still alive and if not restart sshd. Maybe SystemD can so something similar but I am not familiar with SystemD.

Edit: A quick and simple script looks like this

#!/bin/sh

# Check if `sshd` process is running; If not running, `pgrep` returns
# an exit status of '1' and restarts `sshd`
pgrep 'sshd' > /dev/null || systemctl restart sshd

Make the script executable with chmod +x /home/user_name/sshd_check

Add the following line to /etc/crontabs/root to run the script every 15 minutes

 */15    *       *       *       *       /home/user_name/sshd-check

I don't use SystemD but I am pretty sure systemctl restart sshd is correct, otherwise it can be changed to whatever your operating system uses to control services

[–] gedfromgont@piefed.ca 1 points 1 month ago (1 children)

Thanks, I'll try this. What speaks against this is that all my services turn unreachable, and those do not rely on ssh access. But still worth a try.

[–] confusedpuppy@lemmy.dbzer0.com 1 points 1 month ago

Aah you did mention that, my eyes just decided to skip that when I read your post.

This reminds me of another issue I ran into but I use Alpine Linux so I don't know if it's a distribution specific issue. I'll share the issue and workaround solution anyways as something to consider.

Networking on Alpine Linux is controlled by a process called networking and for reasons I don't understand and can't see by any logs, it just stops working. I can't ssh or access the reverse proxy port. I don't remember if ping was working or not as it's been a while since I dealt with it now.

My work around was to have a script on my server ping a known location and restart networking if it couldn't ping out. If a second ping after restarting the process failed, it would then restart the device. This script would run every 15 minutes.

It's a bandage solution that doesn't solve the problem but it does keep my server running. However it seems like pings still work with your server so you might need to get creative in how you test your server's connectivity.