IOError: [Errno 49] Disc quota exceeded: '/opt/splunk/var/run/splunk/session-'

  IOError: [Errno 49] Disc quota exceeded: '/opt/splunk/var/run/splunk/session-'

While logging to any Splunk Instance through web browser

if you encounter the below error on the screen :

IOError: [Errno 49] Disc quota exceeded: '/opt/splunk/var/run/splunk/session-'

First of all you must do the following :

1. ssh <Splunk_Instance> <enter>

2. /opt/splunk/bin/splunk status <enter>

   splunkd is not running.  ----> NOT RUNNING

   splunkweb is running (PID: 14841).

3. Now,your splunkd is not running, you have to start it

   /opt/splunk/bin/splunk start splunkd <enter>

 O/P :- 

Splunk> See your world.  Maybe wish you hadn't.

Checking prerequisites...

        Checking mgmt port [8089]: open

        Checking configuration...  Done.

        Checking critical directories...        Done

        Checking indexes...

homePath='/opt/splunk/var/lib/splunk/XXX/db' of index=_audit on unusable filesystem.

Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

Look at the "Red" marked lines in the above output .

The problem could exist for the XXX index.

4. Now, you go to the index directory :

   cd /opt/splunk/var/lib/splunk/XXX/db <enter>

   

   Delete the unwanted files/directory which are not being used

                            OR 

   Archive them to some other location/directory

5. Now, you should start the splunk :

   /opt/splunk/bin/splunk restart <enter>

Note : Hope whenever you encounter :

IOError: [Errno 49] Disc quota exceeded: '/opt/splunk/var/run/splunk/session-'

the above error, you will be able to solve it immediately !!

Happy Splunking !

  

0

Received event for unconfigured/disabled index...stash ( 1 missing total )

received event for unconfigured/disabled index='xxxx' with source='source::yyyy' host='host::zzzz' sourcetype='sourcetype::stash' ( 1 missing total )

Please find below some of the short cuts being used in the below article :

SH   = Search Head

IDX = Indexer

UF   = Universal Forwarder

HF   = Heavy Forwarder

____________________________________________________________

When you login to your SH or IDX, you will get the following Message Error

on your 'message' tab on the top of your Splunk Page :

received event for unconfigured/disabled index='xxxx' with source='source::yyyy' host='host::zzzz' sourcetype='sourcetype:;stash' ( 1 missing total )


Please find below the two solutions : 


1.  If UF/HF box is connected to the IDX box, and UF/HF
   is configured to send the data to the index="XXXX" 
   in IDX box, but if in IDX box, there is no index="XXXX"
   created or somehow disabled,then the above message appears.

   In this case either you enable/create index="XXXX" in IDX
   box or disable the data forwarding in UF/HF for that
   particular index as shown below :

   ssh <UF> OR <HF> <enter>
   cd /opt/splunk/etc/system/local <enter>
   vi inputs.conf <enter>

   [monitor:///<some_path>]
   index="XXXX"
   
   # Add the below link to disable the input forwarding

   disabled=1

   /opt/splunk/bin/splunk restart <enter>


***************************        ****************************


2.  Go to host by ssh :

    ssh ZZZZ <enter>
    cd /opt/splunk/etc/ <enter>
    find . | xargs grep -i "XXXX" <enter>

    We know that ZZZZ is a SH and in SH we may have saved
    Searches which are using "XXXX" name in their Queries.
    So, we will go to that application and look into the 
    "savedsearches.conf" file.


   cd /opt/splunk/etc/apps/<app_name>/local/ <enter>
   vi savedsearches.conf <enter>
   
   search for "XXXX" and disable that saved search where this 
   index is being used ( Just add the below line ) :

    disabled=1

   OR , you can also disable your saved search from GUI :)

   Go to Splunk SH web interface,
   https://ZZZZ/:8000 <enter>

   Go to Manager --> Searches and Reports  and then search 
   for the Index Name ,all the saved search will show up 
   which are using that index name. Simply go to the 'Status' 
   field and 'Disable' it.


Note : There could be a situation where in the query the index name ( XXXX ) is 
            not there  but in the summary indexing they are using this Index Name.
            Simply click on'saved search' , come down and uncheck
           "Summary Indexing ( Enable )" [ If you see 'XXXX' is written in place ]

Hope whenever you encounter :

received event for unconfigured/disabled index='xxxx' with source='source::yyyy' host='host::zzzz' sourcetype='sourcetype:;stash' ( 1 missing total )

the above error, you will be able to solve it immediately !!

Happy Splunking !

1

Could not send data to output queue (parsingQueue), retrying... ( Part 2 )

Could not send data to output queue (parsingQueue), retrying...

You can increase the file descriptors, etc. but you will probably still have performance issues. I am sure that the forwarder is consuming more CPU and memory than it should, too.

Even if only a portion of these files are actively being updated, Splunk will monitor ALL of them. This means that Splunk will examine the mod time of each file in a round-robin fashion. Over and over again, even though nothing has (and maybe never will) change. Because Splunk can't know which files will or won't be updated.

This is obviously a huge waste of machine time if most of the files are not being updated. 

Here are some steps that you could take:

1. Remove the older files.

1. Rename the older files to a name, perhaps xyz.OLD. Blacklist files using the regex .OLD$ and Splunk will skip them.

1. Use the ignoreOlderThan = <time window> in inputs.conf - but BE CAREFUL. ignoreOlderThan causes the monitored input to stop checking files for updates if their modtime has passed this threshold. So if you set it for 14d, then you can't ever add a file older than 2 weeks into the directory. (Well, you can, but Splunk will ignore it.)

  1. By default the Forwarder limits its use of the network to 256 KBPS to avoid 

     saturating the network .

     You can change this by editing /opt/splunk/etc/system/local/limits.conf:

[thruput] maxKBps = 0 # means unlimited

/opt/splunk/bin/splunk restart <enter>

To check the Part One Solution click on the below link :

               Part One Solutions

Hope you get rid of the below message from being appearing in your log :

Could not send data to output queue (parsingQueue), retrying...

Happy Splunking !!

0

Could not send data to output queue (parsingQueue), retrying...

Could not send data to output queue (parsingQueue), retrying...

The TailingProcessor message means that it was unable to insert data into the parsingQueue, which, as you might guess, is where event parsing happens. This only happens when the queue is full - to confirm this, run 'grep blocked=true /opt/splunk/var/log/splunk/metrics.log*' and you should see many results.

Data travels through Splunk's queues pretty much linearly - meaning that if a queue further down the line (indexQueue) is blocked, it will eventually fill and block the queues that are feeding it (parsingQueue). 

Note :         The metrics.log output will indicate which queues are blocked.

Below are some checks to perform :

Does metrics.log on the indexer (receiver of the forwarded data) also contain blocked=true messages?

• If so, the problem is likely on the indexing side only - perhaps the indexing box is overloaded (slow disks, etc).

If not: does metrics.log on the forwarder indicate that only the parsingQueue is blocked, or indexQueue as well?

• Seeing only the parsingQueue would indicate that something along the lines of too much time spent running complex regexes on the data.
• A blocked indexQueue would mean the data isn't being sent to the indexer fast enough. What type of link is between the forwarders and indexer? How much bandwidth are you actually seeing being used by the forwarding port?

Possible Solution / Common Workaround : 

1. It's certainly possible that the link between your Forwarder and Indexer is having issues.This indicates either network problems OR problems with the config of outputs.conf on the forwarder (to get to the indexer) or inputs.conf in the indexer (to receive from the forwarder).

2. Network port is not opened between your Forwarder and the Indexer

3. Stop the Forwarder remove the metric logs

  from /opt/splunk/var/log/splunk/metrics* and re-start the Forwarder

Splunk starts blocking when free space falls below 2GB and it sounds like this was the cause of your problem. 

Just deleting the files will work as a temporary fix but they will grow back again :)

You can change the default in "/opt/splunk/etc/system/local/server.conf"

[diskUsage]

minFreeSpace = 1000

4. This error may be caused by a mis-configured outputs.conf/inputs.conf.

Ensure the listening port on the Indexer is a configured receiver for spunktcp and not the port of splunkd.

5. We may have to up ulimits on the OS and increase the max_fd in limits.conf

   on the Splunk Forwarder 

6. We can add entries as below in the $SPLUNK/etc/system/local/server.conf file to increase the parsing queue :

  [queue=parsingQueue] 
    maxSize = 10MB

If your problem is not yet solved then click on the below link :

               Part Two Solutions

Hope you get rid of the below message from being appearing in your log :

Could not send data to output queue (parsingQueue), retrying...

Happy Splunking !!

0

Linux transparent hugetables support, enabled="always" defrag="always"

Linux transparent hugetables support,

 enabled="always" defrag="always"

1. Go to your Search Head OR Indexer and open a GUI :

  

     https://<SearchHead_OR_Indexer_URL>:8000  <enter>

    

2.  Go to "Searching and Reporting" Application from the "Apps" Menu on the top left

3. Give the following Splunk Query on the Search Box :

index=_internal (transparent always)

OR recommended sourcetype=splunkd

| stats count by message,

If the output comes as :

Linux transparent hugetables support, enabled="always" defrag="always"

Remedy :

1.  Go to box ( Search Head ) OR ( Indexer ) as 

    ssh <Search_Head_IP> OR <Indexer_IP> <enter>

2.  Check if your OS is affected:

    cat /sys/kernel/mm/transparent_hugepage/enabled <enter>

    O/P : [always] madvise never

    cat /sys/kernel/mm/transparent_hugepage/defrag <enter>

    O/P : [always] madvise never

3.  Fix on OS level:

    echo never > /sys/kernel/mm/transparent_hugepage/enabled <enter>

    echo never > /sys/kernel/mm/transparent_hugepage/defrag <enter>

4.  Validate on OS level:

    cat /sys/kernel/mm/transparent_hugepage/enabled <enter>

    O/P : always madvise [never]

    cat /sys/kernel/mm/transparent_hugepage/defrag <enter>

    O/P : always madvise [never]

Note : Hope whenever you encounter :

Linux transparent hugetables support, enabled="always" defrag="always"

the above error, you will be able to solve it immediately !!

Happy Splunking !

0

The hard fd limit is lower than the recommended value

The hard fd limit is lower than the recommended value.
The hard limit is '4096' The recommended value is '64000'.

1. Go to your Search Head OR Indexer and open a GUI :

  

     https://<SearchHead_OR_Indexer_URL>:8000  <enter>

    

2.  Go to "Searching and Reporting" Application from the "Apps" Menu on the top left

3. Give the following Splunk Query on the Search Box :

index=_internal (transparent always)

OR recommended sourcetype=splunkd

| stats count by message, host

If the output comes as :

The hard fd limit is lower than the recommended value. 

The hard limit is '4096' The recommended value is '64000'.

Remedy :

1. Go to box ( Search Head ) OR ( Indexer ) as 

   ssh <Search_Head_IP> OR <Indexer_IP> <enter>

2.  ulimit -n 64000

3.  vi /etc/security/limits.conf <enter>

    # Add the below two lines in the file "limits.conf" 

    root          soft   nofile         64000

    root          hard   nofile         64000

4.  ulimit -Hn <enter>

    # Output should be '64000' 

Note : Hope whenever you encounter :

The hard fd limit is lower than the recommended value. 

The hard limit is '4096' The recommended value is '64000'.

the above error, you will be able to solve it immediately !!

Happy Splunking !

0