TRANSFORMS-set_source = setDynamicSourceJSON, setDynamicSourcetypeJSON, setDynamicHostJSON TRANSFORMS-set_source = setDynamicSource, setDynamicSourcetype, setDynamicHost The files included the following settings: I created the following app and pushed it out to my Splunk environment: Indexers that will be ingesting the new data (If using a clustered environment, use the Indexer cluster Manager Node, previously Master Server/Cluster Master to push out the configuration).Search Head running the search to collect/re-index the data in a new index (If using a cluster, use the deployer to push out the configurations).The best way to add the props and transforms is to package them up in a separate app and push them out to the following servers: I will outline the process I used for both types of events using the _internal and _introspection index in which you can find both single-line plain text events and single-line JSON events. Use props to apply those transforms on the source.Create Transforms to extract the values of those fields.Append the new fields to the end of the raw event (for JSON, we had to make sure they were part of the JSON blob).Create three new fields in search for source, sourcetype, and host.Make sure the target index is created on the indexers.This test was carried out on a single-instance Splunk server, but if you are doing it in a distributed environment, I will list the steps below on where to install props and transforms. The transforms extract field values based on regex, therefore everything has to be set up with care to make sure the regex works as designed. Note: This method is custom and only works if all steps are followed properly. This method is by no means perfect and only works with two types of events: I chose props and transforms to solve my problem. To solve this problem, I had to get creative, as there was no easy and direct way to do that. ![]() We also discovered that when summarizing raw events, Splunk will not add orig_source, orig_sourcetype, and orig_index to the summarized/re-indexed events. For example, host = host would literally change the host value to “host” instead of the host field in the original event. To our dismay, this was not what happened, as anything added to those fields is treated as a literal string and doesn’t take the dynamic values of the field being referenced. The method we were going to follow was simple – build the search to return the events we cared about, use the collect command with the “sourcetype,” “source,” and “host” values to get the original values of source, sourcetype, and host to show up in the new event. To change these, you can specify these values as parameters in the collect command. When re-indexing raw events using the “collect” command, Splunk automatically assigns the source as search_id (for ad-hoc search) and saved search name (for scheduled searches), sourcetype as stash, and host as the Splunk server that runs the search. ![]() This would enable us to search for the specific event we want and reindex them in a separate index. This was historical data and could not be easily routed to a new index at index-time.Īfter much deliberation on how to move this data over, we settled on the summary index method, using the collect command. Re-Index Raw Splunk Events to a New Indexīy: Zubair Rauf | Splunk Consultant, Team LeadĪ few days ago, I came across a very rare use case in which a user had to reindex a specific subset of raw Splunk events into another index in their data.
0 Comments
Leave a Reply. |