Splunking DataDog Log Events

Introduction

DataDog is a SaaS-based data analytics platform monitoring service for cloud-scale applications that provides monitoring of servers, databases, tools, and services. One of my main issues is the log retention and I’ve found certain limitations on what you can do with reporting and analytics. If we need to do more advanced mathematical/graphing operations, might need to use another BI tool to accomplish.

Splunk allows me to have data search on-demand plus, it has amazing visualizations features that display various charts (not to mention its new Dashboard Studio)

Don’t get me wrong. I’m a DataDog user myself and I love some of its out-of-the-box integrations features (like AWS) and, I know, this tools weren’t built for exactly the same purpose but… what would happen if we can get all the data (from different platforms) connected and have them work together to keep us covered?

Let’s say you are in the middle of an investigation and you want a wider picture for some Splunk events. Using Datadog and Splunk together, further increases visibility into the infrastructure and helps identify the root cause of problems faster and with greater accuracy.

I want to introduce a Splunk Add-On that demonstrates how these great platforms can work together: Simply give this app a DD query and it will pull the data from DataDog to Splunk in a matter of seconds and, it’s 100% Splunk 8.2 compatible.

In my case I’m using a DataDog free 14 day trial and its corresponding agent to send the logs to the Datadog cloud platform. Simply pick the platform or service that you want to monitor, download the agent specific to that device or service, and enter your API key (the key is tied to your account). However, it can also be used by any custom agents developed by a third party (like NXLog). You can find further information about agent installation and configuration here.

Ok, enough intro. Let’s get to it!

Add-On installation

  1. Download DataDog Log Integrator from Splunkbase.
  2. In Splunk go to Manage Apps menu and click on Install app from file.
  3. Select the Add-On file you’ve just downloaded and click on Upload.
  4. You must restart Splunk server to enable the Add-On.

Step by Step Configuration

Once Splunk restarted you will see a new app within the left menu.

Now it’s time to configure our data source. First, click on the Add-On name to access.

Go to Configuration menu and then Add-on Settings tab.

The default DataDog search endpoint has been set to https://api.datadoghq.com/api/v2/logs/events/search

Enter the API-Key and APP-Key previously generated on the DataDog ‘Integrations APIs’​ setup.

Click Save.

Now go to the Inputs menu and then click on Create New Input.

Fill the corresponding form and enter the DataDog query you want to execute programmatically (don’t worry, this Add-on has its own checkpoint to avoid data duplication).

Last, click on Add

Congratulations! you have successfully configure DataDog logs as a Splunk Input.

And lastly it’s time to run some Splunk searches…

Note:

By the way, I’ve previously setup a Splunk SmartStore (S3 bucket) on my personal IBM Cloud Object Storage because… well, why not?!

Also, it might be better to leave this other storage option for another article.

Happy Splunking! 🤓