Home
Forums
New posts
What's new
New posts
Latest activity
Log in
Register
What's new
New posts
Menu
Log in
Register
Install the app
Install
Home
Forums
Other Topics
Off topic discussion
How do I solve this weird server problem?
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Reply to thread
Message
<blockquote data-quote="the Thing" data-source="post: 1204943" data-attributes="member: 9305"><p>Roosh here's your root cause. You're most likely getting DDOSed.</p><p></p><p>Advanced botnets can actually "masquerade" as real browsers. They achieve this via malware installed on unsuspecting victims' machines, by running Chrome/Firefox (or any supporting browser) in something called "headless" mode. Headless browsers work just like full fledged browsers using the same rendering engine, but instead of rendering the page on screen they do it in memory, in the background. Since it's essentially the same as the real thing and it works exactly in the same way (loading the page and all resources and running any javascript on the DOM the same way a real browser would) we use it for things like automating GUI testing. But it can also be used for mischief.</p><p></p><p>Cloudflare will do a pretty good job of mitigating threats like this but it won't be a 100% solution. The spiked metric "browser challenged" means exactly this; it means a browser navigated to your page and managed to get past the initial checks, but then Cloudflare ran some more advanced heuristics (essentially "challenging" the browser) to determine if the visit was legit but since there was no actual human being behind the headless browser, it failed this challenge and got cut off.</p><p></p><p>Memory usage being high could be easily related to the server receiving way too many requests. Then the server process crashing would be due to running out of memory. MySQL most likely isn't your root cause here -- increased request count means more MySQL loads means higher CPU usage.</p><p></p><p>Seeing one UDP IN block every 30 seconds isn't something you should worry about. There are hackers out there that probe whole IP ranges all the way to see if they can find any machines with known vulnerabilities. If you were getting targeted you'd see thousands per second.</p><p></p><p>One thing you should definitely do is to examine the output of "top -bcn1 -w512" here 'b' is batch mode, 'n1' is take 1 sample, 'c' shows the full command line for each process and 'w512' makes the output 512 chars wide so the whole command will wrap around and fit the screen. The full command line for each process is important since it will be easy to tell if there's any fucky process going on. You can post the output, or preferably, post in private or send me over PM (so you dont expose anything about your server filesystem structure publicly) and I'll take a look.</p><p></p><p>Another thing you should definitely do is to check access and error log files, these can be anywhere depending on your distro but try "/var/log/apache2/access.log" and "/var/log/apache2/error.log". If they're not there, maybe try under "/var/log/httpd/" or google your distro name and check where the logs are. The error log is especially important in your case. Check these to see if you can see any sketchy patterns.</p><p></p><p>Apart from that you can look at "last" command output to see last logins. "w" command to see who has a shell open at any given time. See if there are any logins that you don't recognize.</p><p></p><p>Same with syslog. It can be at different paths depending on your distro. Seems yours log at "/var/log/messages" which makes me think you're running something like RHEL/CentOS/Fedora.</p><p></p><p>I don't know what forum/blog software you run (apart from Wordpress) but in the unlikely event that you've been breached, the hacker probably put backdoors there to gain easy access next time around. I think WP has a file integrity checker, you should run it. For an all-around WP security solution I usually install a plugin called "Sucuri" for my setups.</p><p></p><p>Last but not least, keep the server up to date. Best of luck..</p><p></p><p>Edit: I saw that you're running php as the users 'rok' and 'rooshv' are you using these for terminal login by any chance? If yes, that's an ENORMOUSLY huge gaping security hole! (10x as big if they also have sudo access) You should definitely disable their shells and passwords, limit their access and create another limited account to use for SSH'ing into the host. If that's the case let me know and I can help you with that too.</p></blockquote><p></p>
[QUOTE="the Thing, post: 1204943, member: 9305"] Roosh here's your root cause. You're most likely getting DDOSed. Advanced botnets can actually "masquerade" as real browsers. They achieve this via malware installed on unsuspecting victims' machines, by running Chrome/Firefox (or any supporting browser) in something called "headless" mode. Headless browsers work just like full fledged browsers using the same rendering engine, but instead of rendering the page on screen they do it in memory, in the background. Since it's essentially the same as the real thing and it works exactly in the same way (loading the page and all resources and running any javascript on the DOM the same way a real browser would) we use it for things like automating GUI testing. But it can also be used for mischief. Cloudflare will do a pretty good job of mitigating threats like this but it won't be a 100% solution. The spiked metric "browser challenged" means exactly this; it means a browser navigated to your page and managed to get past the initial checks, but then Cloudflare ran some more advanced heuristics (essentially "challenging" the browser) to determine if the visit was legit but since there was no actual human being behind the headless browser, it failed this challenge and got cut off. Memory usage being high could be easily related to the server receiving way too many requests. Then the server process crashing would be due to running out of memory. MySQL most likely isn't your root cause here -- increased request count means more MySQL loads means higher CPU usage. Seeing one UDP IN block every 30 seconds isn't something you should worry about. There are hackers out there that probe whole IP ranges all the way to see if they can find any machines with known vulnerabilities. If you were getting targeted you'd see thousands per second. One thing you should definitely do is to examine the output of "top -bcn1 -w512" here 'b' is batch mode, 'n1' is take 1 sample, 'c' shows the full command line for each process and 'w512' makes the output 512 chars wide so the whole command will wrap around and fit the screen. The full command line for each process is important since it will be easy to tell if there's any fucky process going on. You can post the output, or preferably, post in private or send me over PM (so you dont expose anything about your server filesystem structure publicly) and I'll take a look. Another thing you should definitely do is to check access and error log files, these can be anywhere depending on your distro but try "/var/log/apache2/access.log" and "/var/log/apache2/error.log". If they're not there, maybe try under "/var/log/httpd/" or google your distro name and check where the logs are. The error log is especially important in your case. Check these to see if you can see any sketchy patterns. Apart from that you can look at "last" command output to see last logins. "w" command to see who has a shell open at any given time. See if there are any logins that you don't recognize. Same with syslog. It can be at different paths depending on your distro. Seems yours log at "/var/log/messages" which makes me think you're running something like RHEL/CentOS/Fedora. I don't know what forum/blog software you run (apart from Wordpress) but in the unlikely event that you've been breached, the hacker probably put backdoors there to gain easy access next time around. I think WP has a file integrity checker, you should run it. For an all-around WP security solution I usually install a plugin called "Sucuri" for my setups. Last but not least, keep the server up to date. Best of luck.. Edit: I saw that you're running php as the users 'rok' and 'rooshv' are you using these for terminal login by any chance? If yes, that's an ENORMOUSLY huge gaping security hole! (10x as big if they also have sudo access) You should definitely disable their shells and passwords, limit their access and create another limited account to use for SSH'ing into the host. If that's the case let me know and I can help you with that too. [/QUOTE]
Insert quotes…
Verification
Post reply
Home
Forums
Other Topics
Off topic discussion
How do I solve this weird server problem?
Top
