cross-posted from: https://piefed.ca/c/selfhosted@lemmy.world/p/957754/local-homeserver-suddenly-not-responsive-to-ssh-but-can-be-pinged
Cross-posting this from Lemmy world where it seems my post coming from piefed.ca is not federating.
This is something that seems to happen randomly to my little RaspberryPi 4B homeserver, like once every few months maybe.
What happens is that out of nowhere (meaning I didn’t make any change to config or installation) I cannot reach any services I host on it (i.e. Jellyfin, Anchor notes…). I cannot access it via SSH at all, connection keeps timing out.
Ping however gives a signal. I can see/hear its running and doing something, since I hear the external HDD working. It is running headless and I have no monitor and/or keyboard I could connect.
So far my only way out is to just switch its power off, which is obviously not good for the hard drive. It has happened now like half a dozen times over the past few years, so I am wondering what a better way is to handle this. This was on RPI OS and now its happening on DietPi, so seems not related to the OS version.
I tried sending a shutdown signal via SSH but that also just times out, so how do I get it to shutdown properly?
Secondly, what kind of logs could I look at or start collecting to make sense of this?
This happened when my sshd dameon did not start or died. You could use systemctl status to check it.
It has also happened when my white listed static IP for my machine that I access the server with changed. Do you have a default DROP policy?
Timing out indicates that either the sshd service doesn’t exist or is unreachable (but not blocked).
Check the dmesg for all sshd notifications and maybe you find why it is crashing (if the service is dead)
This can sometimes happen if the filesystem has failed or is otherwise inaccessible. SSH needs to read config files and if it can’t, it fails to connect. It would try to write a failure to the log, but can’t.
Things that don’t require filesystem access, and can run totally from processes in memory interacting with hardware, would be fine. So your Ethernet stack works, and so does ping. A serial port would probably work. SNMP would work, too, for anything that doesn’t need to get a metric from the filesystem. I bet if you had a remote syslog server, you would get the logs that are falling out of your filesystem.
I think you might onto something with the filesystem. I am using Hd-idle to switch off the HDD when not in use. It was of course in use the entire time during this, but I noticed in the logs that it came up multiple times as “assuming disk spun up after long sleep” even though it hadn’t been off at all.
Only saw this after I made my post and I cannot make sense of it, but I have never seen this message otherwise, it only happened during this period of not being able to access the server.
I have also seen a situation where ssh did not accept a connection anymore when the disk was full.
Nope, disk is far from full. Like 30% at the moment.
I had a similar situation. Turned out to be a grounding issue.
Kept randomly losing the root m.2 drive and when I’d plug in monitor/keyboard I could type in the terminal, but most executables would not work.
Hard reboot and it’d be good for a few weeks or few months. Then it’d die again. Really annoying losing it because my calendar and email gui is on it.
Once I noticed a small static shock when I touched the case.
I taped a wire to the aluminum case and taped the other end to a grounded chassis of another computer and all was good since then
Interesting. Not sure this is it, definitely didn’t feel a shock the last few times I handled the case during this. But thanks for sharing! I’ll keep it in mind.
You might be somehow locking your self out with fail2ban.
I am not aware I have that installed/configured, not sure if this comes with a DietPi installation?
If we assume yes, how would I fix this?
Edit: looked around online, is there supposed to be a /etc/fail2ban directory? If so, I don’t have it so I guess this is not installed.
This was my first thought
Like a spinning disc drive? I always found those separately powered drives flakey on a PI, the drive was fine, the pi is fine, the dock I used was solid on my PC and laptop, but not the pi.
Is it your OS drive? Could you get a tiny USB SSD and try that out as primary?
The OS is on a SD card, the HDD is a spinning disc connected via USB and I actually never had any issues with its connectivity.
There are however issues sometimes if I connect a second drive, where I assume it does have something to do with not getting enough power for 2 external HDDs, but I only ever connect a second drive for backing up.
Don’t think this is connected, but if more people here feel like yes then I am happy to check in that direction.
Are you powering the spinning disc off the pi? High draw can cause funky stuff too.
Any under voltage warnings?
Yes, disk is powered via USb. I had however a different disk before that was separately powered and I think I also ran into this issue.
Where would I see under voltage warnings? I kind of doubt this is it because for months everything is fine, this sounds like it would cause issues more regularly.
The under voltage might just be in the TTY 1 or whichever is the main console when you plug a monitor in.
I think it’s two different issues that are compounding, one being the maybe a bit too high power draw from the exterior drive, and the second being the different power sources being funky. The vexing part was the infrequency.
Sorry I can’t be more useful with diagnostics, my problems all went away when I went with a low power USB drive IE a USB SSD.
Oh no worries. I wrote this post with no expectations whatsoever since I myself have no clue about this.
I’ll make a note on checking power consumption, also when I am planning to buy the next drive, which won’t be for a while though. Until then I will just make sure to do regular backups.
last time that happened I had updated SSHD and caused a mismatch in the list of provided/accepted ciphers and I ended up needing to restart
just my 2 cents
Do you have a display and keyboard you can connect to it? If you can get video out, and it accepts keyboard input, you can just log in locally and reboot.
I have a desktop system that crashes similarly, though pings also fail. Unfortunately for me, local access also doesn’t work so I simply hard power cycle it like what you’ve been doing for your Pi.
It is running headless and I have no monitor and/or keyboard I could connect.
From my post. But also it sits in a shelf where anyway don’t have much space to put anything else. It also happens too rarely that it wouldn’t make sense to get specifically for this a display and keyboard in place.
Oh apologies, I missed that line in your post.
You wouldn’t need to keep the monitor/keyboard there the whole time. Just connect them temporarily when it has the issue. This obviously doesn’t work if you don’t have a monitor or a keyboard you could borrow from a different system.
From what machine? Pc? Mobile? If a PC, how is it connected to the Pi? Direct cable, a switch, a WiFi…? Does the Pi use a fixed address in your home network?
From phone or laptop, doesn’t matter. The Pi is in my network via Wifi, well technically its connected to the router via cable but then everything else is connecting via Wifi through the router to the Pi. I have given the Pi a fixed IP.
