Followers

Sunday, January 31

Splunk Search Processing Language ( Commands for Developers )

0

Architecture of Splunk

1

Thursday, May 14

Usage of Splunk EVAL Function : SEARCHMATCH

                                         Usage of Splunk EVAL Function : SEARCHMATCH


       Returns true if the event matches the search string X.




      Find below the skeleton of the usage of the function "searchmatch" with EVAL : 

                                               searchmatch(X)




index=_internal 
| eval AA=if(searchmatch("Queue Full"),"Exists","NOT")






Explanation :

             If any event in the "_internal" index is having 
             "Queue Full" String in it, then "Exists" will be 
             stored in the "AA" field which is newly created.
             If not then "NOT" will be stored in the "AA"

             So, there are total 5990 lines with "Queue Full
             and 5452 without "Queue Full"
           


Verification :


   index=_internal 
   | eval AA=if(searchmatch("Queue Full"),"Exists","NOT")
   | search AA="Exists"


Only those lines will appear which has "Exists" value in AA field,
which means "Queue Full" string is there in the event.







Similarly , you can also verify for AA="NOT" and you will get 
only those lines which do not contain "Queue Full" String in it.



Now you can effectively utilize "searchmatch" function with "eval" command to meet your requirement !!


Hope you are now comfortable in :
Usage of Splunk EVAL Function : SEARCHMATCH


HAPPY SPLUNKING !!
2

Effective Usage of "STRPTIME" and "STRFTIME"

Effective Usage of "STRPTIME" and "STRFTIME" 


Below is the effective usage of the "strptime" and "strftime
function which are used with eval command in SPLUNK :



1. strptime() :
                It is an eval function which is used to 
                parse a timestamps value


2. strftime() :
                It is an eval function which is used to 
                format a timestamps value



Let's say you have a timestamps field whose value is like :

1. 13/May/2015:15:32:11.410 +0000
213/Jul/2014:15:31:48.387 +0000   and so on ...


and we want the output like :

1. 20150513
2. 20140713



Below examples will show the real usage of "strptime" and "strftime"


you have to make a two stage operations, first convert your input format to "epoch" and then convert it to your desired format.


1.  index=_internal sourcetype=splunkd_access
    | rex field=_raw ".*\[(?P<NEW_FIELD>.*)\].*" 
    | table NEW_FIELD 
    | eval FIELD=strptime(NEW_FIELD,"%d/%b/%Y:%H:%M:%S")


                                NEW_FIELDFIELD
  13/May/2015:15:49:41.308 +00001431532181.000000
  13/May/2015:15:49:36.308 +00001431532176.000000
  13/May/2015:15:49:32.553 +00001431532172.000000
  13/May/2015:15:49:32.544 +00001431532172.000000
  13/May/2015:15:49:32.537 +00001431532172.000000
  13/May/2015:15:49:32.528 +00001431532172.000000
  13/May/2015:15:49:32.518 +00001431532172.000000


Explanation : 

             "NEW_FIELD" is an existing field which has a value
              as shown above.
              "strptime" function converts the value of
              "NEW_FIELD" to "epoch" and stores in a newly 
              created variable called "FIELD"


Note : If you time is "2015-03-27T15:49:34Z" then
       strptime would be "%Y-%m-%dT%H:%M:%SZ"


Now, in order to get the Desired Output in a right format
use "strftime" function on the "epoch" value , i.e., "FIELD"



index=_internal sourcetype=splunkd_access 
| rex field=_raw ".*\[(?P<NEW_FIELD>.*)\].*" 
| table NEW_FIELD 
| eval FIELD=strptime(NEW_FIELD,"%d/%b/%Y:%H:%M:%S") 
| eval DesiredTime=strftime(FIELD,"%Y%m%d") 
| fields - FIELD


           NEW_FIELDDesiredTime
      13/May/2015:15:59:36.247 +000020150513
      13/May/2015:15:59:31.540 +000020150513
      13/May/2015:15:59:31.247 +000020150513
      13/May/2015:15:59:29.355 +000020150513
      13/May/2015:15:59:28.896 +000020150513


Explanation : 

              "DesiredTime" is the newly created field which is 
               using "strftime" function to format the "epoch"
               time to its desired format.



If splunk has read your timestamps(without the year) and parsed
and indexed it correctly( You can always compare the timestamps in the events with the timestamps next to the blue down-arrow to the left of the event ), then you can skip the first part ( strptime )
and use the _time field, which is already in epoch.


index=_internal
| eval DesiredTime=strftime(_time,"%Y%m%d") 
| table _time , DesiredTime



_timeDesiredTime
2015-05-14 10:35:1620150514
2015-05-14 10:35:1620150514
2015-05-14 10:35:1620150514
2015-05-14 10:35:1520150514



So, Finally you have got an idea how to do "Effective Usage of "STRPTIME" and  "STRFTIME"


Happy Splunking !!






3

Monday, May 11

Advantage of Using "Splunk Light" for the Splunkers in the Organization

Advantage of Using "Splunk Light" for the Splunkers in the Organization



We have been using SPLUNK Enterprise version for quite a long time
and we know the real power of it giving Big Data Solutions.

Splunk Enterprise version is used in a very large scale industry,
but what if there is a budget constraint for any industry OR
a small scale industry could not afford the License but still 
they want to avail such a beautiful features of SPLUNK for 
Log Analysis and other stuff.

YES, you are right , Now we have to think of something which can
reduce the problem of Small Scale Industry and that's why 
the "SPLUNK LIGHT" was born !!


Definition :

Splunk Light is a comprehensive solution for small IT environments that automates log search and analysis. It speeds tactical troubleshooting by gathering real-time log data from your distributed applications and infrastructure in one place to enable powerful searches, dynamic dashboards and alerts, and reporting for real-time analysis—all at an attractive price that will fit your budget.

Splunk Light gives you the ability to pro-actively analyze problems and take immediate action without having to manually gather, organize and sift through gigabytes of log data.

Advantages :

1. A comprehensive product for log search and analysis
   built on Splunk’s proven technology

2. Search, Report and Alert on all your log data in 
   Real Time from one place

3. Designed for small IT environments with free download, 
   easy set up and rapid data on-boarding

4. Online purchase starting at $75 per month (Billed Annually)

5. Easily upgrade-in-place to the Full Splunk Enterprise

Why Splunk Light ?

1. Easy to Buy, Deploy and Use :

         Buy online, install and start using in
         just 10 minutes without any system configuration.

2. Full-Featured Log Search and Analysis :

         Everything you need to troubleshoot and secure 
         your IT environment. Includes collection
         indexing, monitoring, reporting and alerting.


3. Priced & Packaged for Small IT Environments :

         Easy download optimized for use on a single server.
         Starts free up to 500MB/day and affordably 
         priced as you grow. 

4.   Built on Proven Splunk Technology :

         Leverage powerful Splunk technology with
         universal collection and indexing from any
         log format and real-time search. 

5. Integrated Monitoring and Alerting :

         Alerts can automatically trigger actions 
         to send automated emails, execute remediation scripts,
         or post to RSS feeds.



 Splunk Light VS Splunk Enterprise Comparison







 Hope you have got an Idea about the SPLUNK LIGHT , its uses,
 its advantages of being used by the Industry very rapidly
 and Advantage of Using "Splunk Light" for the Splunkers 
 in the Organization


Happy Splunking !!


1

Counting of a Particular Character in a Field

Counting of a Particular Character in a Field


     There are many ways to achieve the above scenario :


   1. Using "mvcount and split"  


      index="_internal" 
      | head 4 
      | eval Var="www.google.com" 
      | eval Result=(mvcount(split(Var,"."))-1) | table Var,Result


VarResult
                                                                     www.google.com2
                                                                     www.google.com2
                                                                     www.google.com2
                                                                     www.google.com2



Explanation :

             In the above Example,We are trying to count the
             number of "." ( dots ) in the field "Var".
             With the help of "split" function we have split
             the words by ".split(Var,"."), so there 
             are three words coming "www","google" and "com"   
             and then we are taking the count of these words.
             mvcount(split(Var,".")). So,there are three words.
             But in order to achieve the output we have to 
             subtract "1" from the whole output.
             (mvcount(split(Var,"."))-1) whose result is stored
             in a newly created field called "Result"




Hope this has helped you in achieving the below requirement without fail :

Counting of a Particular Character in a Field


Happy Splunking !!



0

IOError: [Errno 49] Disc quota exceeded: '/opt/splunk/var/run/splunk/session-'


  IOError: [Errno 49] Disc quota exceeded: '/opt/splunk/var/run/splunk/session-'



While logging to any Splunk Instance through web browser
if you encounter the below error on the screen :

IOError: [Errno 49] Disc quota exceeded: '/opt/splunk/var/run/splunk/session-'


First of all you must do the following :


1. ssh <Splunk_Instance> <enter>


2. /opt/splunk/bin/splunk status <enter>

   splunkd is not running.  ----> NOT RUNNING
   splunkweb is running (PID: 14841).



3. Now,your splunkd is not running, you have to start it

   /opt/splunk/bin/splunk start splunkd <enter>


 O/P :- 

Splunk> See your world.  Maybe wish you hadn't.

Checking prerequisites...
        Checking mgmt port [8089]: open
        Checking configuration...  Done.
        Checking critical directories...        Done
        Checking indexes...

homePath='/opt/splunk/var/lib/splunk/XXX/db' of index=_audit on unusable filesystem.
Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue



Look at the "Red" marked lines in the above output .
The problem could exist for the XXX index.


4. Now, you go to the index directory :

   cd /opt/splunk/var/lib/splunk/XXX/db <enter>
   
   Delete the unwanted files/directory which are not being used
                            OR 
   Archive them to some other location/directory


5. Now, you should start the splunk :

   /opt/splunk/bin/splunk restart <enter>



Note : Hope whenever you encounter :


IOError: [Errno 49] Disc quota exceeded: '/opt/splunk/var/run/splunk/session-'


the above error, you will be able to solve it immediately !!

Happy Splunking !


  
0