-
VersionOther (please provide detail below) Installation MethodSecurity Onion ISO image Descriptionupgrading Installation TypeDistributed Locationon-prem with Internet access Hardware SpecsExceeds minimum requirements CPU32 RAM128GB Storage for /500GB Storage for /nsm3T Network Traffic Collectionspan port Network Traffic Speeds1Gbps to 10Gbps StatusYes, all services on all nodes are running OK Salt StatusNo, there are no failures LogsNo, there are no additional clues DetailHi, After upgrading (i.e. 2x soup on the manager node and waiting for all nodes to get upgraded) I went through the steps described in 
"local-rules" points to a file in Also a full update of suricata had no errors. I also "emptied" the local-rules file by running When extending the timeframe for the Alerts display the last alerts have a pre-2.4.201-upgrade-timestamp. Again, Grid says everything is OK as well as "so-status" indicating no errors whatsover. Likewise, PCAP works, i.e. packets are coming in via the sensor node. To me it looks like S.O. completely forgot to send traffic through Suricata after upgrading to 2.4.201. Any ideas on how I can get Suricata working again? Guidelines
|
Beta Was this translation helpful? Give feedback.
Replies: 8 comments 25 replies
-
|
I have had the same problem since 2.4.200 on a SO grid with 19 VM's. I've tried lots of stuff to get alerts working again, no luck. For a while I thought it had something to do with idstools not being properly removed, but it looks like the docker was removed, just not removed from the list of so-status items. I found another post on here mentioning that we can just manually remove that from the status list, which I did. I also followed the post-install instructions regarding the block file and custom rules. No custom rules in our deployment. (nothing but readme files in the custom-local-repos). All indications are a healthy grid now. Green "OK" status on Elastalert, Strelka, and Suricata. CPU utilization on the grid hasn't changed much before and after the update... It seems like the grid is still processing incoming eventlog and network traffic data, just not generating (or displaying) alerts for it in the web interface. Dashboard is full of activity. I have other systems in place to monitor security that meet our requirements, the SO Grid is our way of going above and beyond the bare minimum with more extensive monitoring and log aggregation, and it does a great job at that. I kinda assumed something went wrong during the update that broke something and have been hoping I'm not alone as it's probably too big of a problem to unravel for a single grid deployment. I was figuring on just rebuilding the grid here soon if I didn't see anyone else reporting similar problems. I'm seeing various posts here and elsewhere of similar issues since the 200 release. Hopefully someone figures it out! I suspect many of us are having the same problem, and it's probably a simple fix, we just have to figure out what setting we were all running before the 200 update that isn't compatible with the latest version of SO. I'll keep an eye on this thread. If I find a solution, I'll try to share. (unlikely, as this system is very complicated and I manage dozens of other complex systems, I can only be about ankle deep with any of them). |
Beta Was this translation helpful? Give feedback.
-
|
Are there rules in In Detections are there Suricata rules? On your sensor, do you see Suricata logs in /nsm/suricata? Is your Elastic agent running and healthy on the sensor? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, First of all - thanks very much for looking into this, esp. since this problem brings down one of the main functions of SecurityOnion, i.a. alerting in case something weird happens on the network.
Yes, both nodes (manager and sensor) have the all-rulesets.rules file in their corresponding places: manager-node: sensor node:
Yes, around 80.000 
Oops, looks kinda empty ... but suricata is running: Please let me know if you need any additional logs, traces, etc. in order to track down the underlying problem. |
Beta Was this translation helpful? Give feedback.
-
|
I had a look at our grid to compare - very similar situation. |
Beta Was this translation helpful? Give feedback.
-
|
Well.. I found something: 
My grid was set to default value, but that path doesn't exist. I switched the value to the one shown in the screenshot above and am seeing if that makes any difference. |
Beta Was this translation helpful? Give feedback.
-
|
Red herring... the grid produced a "elastalert missing" error after this change, I tried increasing heap and memory allocations to see if that could be the cause at that point to no avail. Back to default on that setting. For good measure, I did create that directory, but that also makes no difference. On to the next observation: 
"so_detection.is.reporting" is disabled for ALL rules on our Onion. Not sure how this happened, or if it's supposed to be that way? |
Beta Was this translation helpful? Give feedback.
-
Too bad that didn't help :-( Seems we'll have to wait till hopefully someone from SecurityOnion comes up with an idea as to what's going on after the upgrade to 2.4.200 and how to fix this problem, esp. since this problem kills the most basic feature of a NIDS - alarms - completely. |
Beta Was this translation helpful? Give feedback.
-
|
Two additional "observations":
BTW, a |
Beta Was this translation helpful? Give feedback.
-
|
Do you have a custom rules file configured in your suricata.yaml? When you restart Suricata, and check the /opt/so/log/suricata/suricata.log what rules is the service trying to load? Do you see the following file - On your node running Suricata, what do these show? As far as troubleshooting, the steps would look something like below: |
Beta Was this translation helpful? Give feedback.
-
|
Interesting... well I'm guessing my problem is different: cd /opt/so/saltstack/local/salt/suricata/rules/lsall-rulesets.rules {"message":[{"id":0,"rules_loaded":49787,"rules_failed":0,"rules_skipped":0}],"return":"OK"} 
|
Beta Was this translation helpful? Give feedback.
-
|
On the manager node:
No, haven't even touched
On the manager there's no suricata at all - not even any "so-suricata-" command
Yes, it's there: Now for the sensor node that went through the upgrade 190 > 201:
but when I take a look at
which makes sense since there's no Back to Seems like the modifications I made to existing rules via Same for custom variables defined via Last line in These messages correspond to the status for the node on the GUI: 
(NB: Interstingly enough even the full sync takes only a couple of seconds which is very uncommon since it took some minutes )
On the sensore node it's in On the manager node it's in which has a timestamp of when I triggered the manual full sync
Not sure what you ment by "apply the suricata state" but the (again the "all-rulesets.rules" has a timestamp as of yesterday and is different in size compared to the
Here's the full output of |
Beta Was this translation helpful? Give feedback.
-
Just to be clear: The checks you mentioned are to be done/run on the manager or sensor node? |
Beta Was this translation helpful? Give feedback.
-
|
Docker commands on sensor nodes. Troubleshooting steps on manager node. |
Beta Was this translation helpful? Give feedback.
-
Did you remove the block file? When upgrading to .200 there is a syncblock file that will stop alerts. Might have missed it if you referenced it already. |
Beta Was this translation helpful? Give feedback.
-
|
Yes |
Beta Was this translation helpful? Give feedback.
-
|
Yes, same here - as by the book (https://docs.securityonion.net/en/2.4/nids.html#sync-block) ;-) |
Beta Was this translation helpful? Give feedback.
-
|
Did the tests you mentioned - see above #15429 (reply in thread) |
Beta Was this translation helpful? Give feedback.
-
|
Here we go: 
then started a full suricata sync 
which went OK 
...waited around 10 minutes and ... 
on all sensors and alerts are also coming in. So happy to see alerts again :-) BTW, does 47456 (see screenshot above) seem a reasonable number wrt suricata rules loaded? |
Beta Was this translation helpful? Give feedback.
-
|
Looks like somehow my custom rules defined in a rules file (see #15429 (comment)) aren't picked up since I'm again getting alerts for connections I've defined "pass" rules in the custom rules file a long time ago already |
Beta Was this translation helpful? Give feedback.
-
|
Found a possible explanation/reason: 
This is where my custom Suricata rules-file is actually stored On the other hand 
and this directory only holds So here are my questions:
|
Beta Was this translation helpful? Give feedback.
-
|
All rulesets configurations with the changes to Detections should be done via the new consolidated rulest option in |
Beta Was this translation helpful? Give feedback.
-
|
Ah, that explains a lot... Since I had my custom rules in Anyway, just moved my rules file to the new place, did a manual suricata full sync, waited around 15mins and now finally all false alarms are gone while other alarms still come in making my S.O. happy again (finally it's "make your adversaries cry" again instead of "makes the admin cry" ;-) ). Thanks alot for your help over the past days/weeks in getting my S.O. installation working again. Two final questions in this context: As for the Is just one rules file allowed in |
Beta Was this translation helpful? Give feedback.
-
|
In For multiple rules files: Configure it according to - |
Beta Was this translation helpful? Give feedback.
-
|
Alerts stated working again today.... Last night before I left work, I was just going through various settings in the admin>configuration settings, looking for what may be causing the hangup and found that my global>pcapengine was set to transition mode. I believe this was based on a SO guide/doc I was reading some months ago about this becoming the default for regular grids. I decided to switch it back to STENO for now. Came in this morning and alerts are working again. Perhaps it's not supposed to be in transition mode during updates? I wonder... |
Beta Was this translation helpful? Give feedback.
-
|
Hi, It really seems to have something to do with suricata not kicking in on a global basis. |
Beta Was this translation helpful? Give feedback.
-
|
Yea... I went from getting no alerts, to getting SOME alerts by switching that, but none of them are network traffic related, and all but 1 of my sensors now say "0 days pcap retention" and has no data in the suripcap directories in nsm. |
Beta Was this translation helpful? Give feedback.
-
|
Looks like I finally figured out what was causing my problem.. Turns out, somewhere along the way, the bond0 interfaces on the sensors assume MTU 9000. Not sure if this is a change that took place in an update or what, but the virtio interface I have attatched to these virtual machines were set to 1500. After changing the linux bridges on proxmox, and the interfaces on the sensors listening to those bridges ("hub-mode" aging 0) to MTU 9000, the sensors are gathering data again and we're getting alerts again. Works in Suricata or Stenographer mode. |
Beta Was this translation helpful? Give feedback.
-
|
@MDCAllan Please see the note on the Proxmox page in our documentation (https://securityonion.net/docs/proxmox.html):
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks Doug, Yea we're hosted on proxmox (updated to 9 back in Sept), but this had been working fine up until maybe 6-8 weeks ago? There's no physical interface this connects to.. the sensor node just lives on the same physical server with a virtualized firewall... both attached to a virtual bridge (with no-aging set).... Once I set the bridge and virtio interface on proxmox to MTU 9000, everything started working again. The bond0 was able to pick up the assigned interface as a slave. (before it was failing). |
Beta Was this translation helpful? Give feedback.
All rulesets configurations with the changes to Detections should be done via the new consolidated rulest option in
soc > config > server > modules > suricataengine > rulesetSources > default. The other setting will be updated with a deprication warning and changed to readonly.