Monday 22 April 2013

Syslog-ng server recording remote syslog streams

Just a quick how-to for capturing remote syslog streams on Gentoo using syslog-ng

1) Backup your current configuration:
cp /etc/syslog-ng/syslog-ng.conf /etc/syslog-ng/syslog-ng.conf.backup

2) Modify the existing configuration file to accept any and all syslogs directed at the system (note: not exactly 'organized', but useful for a quick replication):

Paste the following above 'source src {' - modify tcp port/max-connections if required:
source s_syslog { unix-stream("/dev/log");
        udp();
        tcp(ip(0.0.0.0) port(5000) max-connections(300));
        pipe("/proc/kmsg");
        };

destination d_syslog { file("/var/log/remote-syslog"); };
log { source(s_syslog); destination(d_syslog); };
3) Restart syslog-ng via openrc (or systemd, whatever you're using). Ignore the pipe/FIFO errors:
 # /etc/init.d/syslog-ng restart
 * Stopping syslog-ng ...
[ ok ]
 * Starting syslog-ng ...
[ ok ]
4) Check to see that your new syslogs are being recorded to /var/log/remote-syslog:
# tail -f /var/log/remote-syslog
Apr 22 15:10:10 NiXToP syslog-ng[8878]: syslog-ng starting up; version='3.3.5'
Apr 23 02:40:52 192.168.0.211 RT_FLOW: RT_FLOW_SESSION_DENY: session denied 192.168.0.10/44870->172.19.1.2/443 junos-https 6(0) global_drop_all(global) Trust Trust UNKNOWN UNKNOWN N/A(N/A) fe-0/0/6.0 No
Apr 23 02:40:56 192.168.0.211 RT_FLOW: RT_FLOW_SESSION_DENY: session denied 192.168.0.10/46526->172.19.1.2/80 junos-http 6(0) global_drop_all(global) Trust Trust UNKNOWN UNKNOWN N/A(N/A) fe-0/0/6.0 No 
For future reference:
# uname -r && syslog-ng -V | head -n 1
3.8.6-gentoo
syslog-ng 3.3.5

Saturday 20 April 2013

Juniper SRX - Determine exact cause of high CPU on PFE

Hi Everyone,

Instead of relying on the sanitized/basic information the SRX generally provides you via usual commands, it's possible to log into the PFE directly and determine which underlying system is causing an issue:

Log into your PFE (there are LOTS of hidden commands available here, but I'll focus on threads):

root@SRX210_A# run start shell pfe network fwdd


BSD platform (OCTEON processor, 416MB memory, 8192KB flash)

FLOWD_OCTEON(SRX210_A vty)# show threads
PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------  ---------------------
  1 H  asleep    Maintenance            976/73824  0/7/14 ms  0%
  2 L  running   Idle                  1296/73824  0/8/10035 ms  0%
  3 H  asleep    Timer Services        1040/73824  0/8/270 ms  0%
  5 L  asleep    Ukern Syslog           856/73824  0/0/0 ms  0%
  6 L  asleep    Sheaf Background      1296/73824  0/0/0 ms  0%
  7 M  asleep    mac_db                 856/73824  0/0/0 ms  0%
  8 M  asleep    Docsis                1072/73824  0/8/184 ms  0%
  9 M  asleep    ATMX                  1136/73824  0/8/491 ms  0%
 10 M  asleep    XDSL                  1352/73824  0/8/30394 ms  0%
 11 M  asleep    DSX50ms               1272/73824  0/8/2491 ms  0%
 12 M  asleep    DSXonesec             1048/73824  0/8/104 ms  0%
 13 M  asleep    SFP                   1216/73824  0/8/295 ms  0%
 14 M  asleep    Ethernet              1184/73824  0/15/90202 ms  1%
 15 M  asleep    RSMON syslog thread   1024/73824  0/0/0 ms  0%
 16 L  asleep    Syslog                1264/73824  0/8/113 ms  0%
 17 M  asleep    Fwdd Notif Recv       1512/73824  0/8/2816 ms  0%
 18 M  asleep    Forwarding Thread     2456/73824  0/15/36745 ms  0%
 19 M  asleep    Periodic             12936/73824  0/15/21958 ms  0%
 20 M  asleep    USB Thread            1328/73824  0/8/776 ms  0%
 21 M  asleep    FPC_IPC-Thread        2368/73824  0/0/0 ms  0%
 22 H  asleep    TTP Receive           1320/73824  0/15/6219 ms  0%
 23 H  ready     TTP Transmit          1104/73824  0/8/1480 ms  0%
 24 M  asleep    UDP Input              904/73824  0/0/0 ms  0%
 25 H  asleep    TCP Timers            1264/73824  0/8/312 ms  0%
 26 H  asleep    TCP Receive            816/73824  0/0/0 ms  0%
 32 L  asleep    Console               3504/73824  7/7/7 ms  0%
.................etc

You can then look into a certain hook more closely if desired:

FLOWD_OCTEON(SRX210_A vty)# show threads 14
PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------  ---------------------
 14 M  asleep    Ethernet              1184/73824  0/15/95814 ms  1%

Wakeups:
      Type  ID  Enabled  Pending   Context
 Semaphore  00      Yes       No  0x489ac008
     Timer  00      Yes       No  0x489ac158

Frame 00: sp = 0x48cbab38, pc = 0x08014d94
Frame 01: sp = 0x48cbabb0, pc = 0x08215c98
Frame 02: sp = 0x48cbabf0, pc = 0x0802b960
Frame 03: sp = 0x48cbac18, pc = 0x00012060


Juniper SRX - Compare Rollbacks

Hi Everyone,

As requested, here is how you can do a diff between rollbacks:

A) Determine which rollback you want to compare via:
root@SRX210_A> show system commit                              
0   2013-04-21 08:46:04 UTC by root via cli
1   2013-04-21 08:44:55 UTC by root via cli
2   2013-04-21 08:43:58 UTC by root via cli
3   2013-04-21 08:42:27 UTC by root via cli
4   2013-04-21 08:35:52 UTC by root via cli
5   2013-04-21 08:33:21 UTC by root via cli
6   2013-04-21 08:25:22 UTC by root via cli
7   2013-04-21 08:24:50 UTC by root via cli
8   2013-04-21 07:57:11 UTC by root via cli
9   2013-04-21 07:56:03 UTC by root via cli
10  2013-04-21 07:53:25 UTC by root via cli
11  2013-04-21 07:52:37 UTC by root via cli
12  2013-04-21 07:47:42 UTC by root via cli
13  2013-04-16 20:25:01 UTC by root via cli

B) Select the rollback you wish to compare based on the above list via:

root@SRX210_A> show system rollback compare 5 0  
[edit security policies from-zone Trust to-zone Trust policy Trust_to_Trust match]
-      source-address Net_192.168.0.0/24;
-      destination-address any;
-      application any;
+      source-address Net_192.168.0.0/24;
+      destination-address any;
+      application junos-icmp-all;
[edit security zones security-zone Trust address-book]
       address Net_192.168.0.0/24 { ... }
+      address Host_192.168.0.10 192.168.0.10/32;

The output above compares rollback '5' to the active configuration '0'




Juniper SRX - Create Global Drop/Cleanup rule

Hi everyone,

To avoid having to create a drop rule with logging enabled on an SRX for everyone Zone-to-Zone possibility, you can now create a global cleanup rule as of 12.1 like so:

# run show configuration security policies global | display set
set security policies global policy global_drop_all match source-address any
set security policies global policy global_drop_all match destination-address any
set security policies global policy global_drop_all match application any
set security policies global policy global_drop_all then deny
set security policies global policy global_drop_all then log session-init

A 'show security policies' will then show this as the last rule:
Global policies:
  Policy: global_drop_all, State: enabled, Index: 8, Scope Policy: 0, Sequence number: 1
    Source addresses: any
    Destination addresses: any
    Applications: any
    Action: deny, log

Looking at the log itself it'll show up as:

Apr 21 08:36:23  SRX210_A RT_FLOW: RT_FLOW_SESSION_DENY: session denied 192.168.0.10/33514->172.19.1.1/80 junos-http 6(0) global_drop_all(global) Trust Trust UNKNOWN UNKNOWN N/A(N/A) fe-0/0/6.0 No

Logging is currently configured as:
set system syslog file traffic-log any any
set system syslog file traffic-log match RT_FLOW_SESSION