Building an Integration for the DataDog Marketplace

Are you Planning to Build a DataDog Integration ? Maybe My Experience can Answer Some Questions.

(Not an official DataDog post)

Intro

A few years ago in 2018 I began using DataDog’s Observability platform in the workplace. I noticed the simplicity and intuitive UI and said ‘…alright not bad’ . It wasn't long until I dealt with their customer service and thats when I realized this company had something special going on. I invested as soon as they went IPO in 2019. They have been growing steadily like I knew they would. I was recently motivated to build an integration for a market that they have yet to address. Before I started the process I was a bit unaware of what I was going to encounter. I am writing the post I wish I had read when I started my journey a few months ago.

Background

There are tons of DataDog integrations out of the box that allow customers to instrument popular cloud apps, databases and other SaaS products. They created a really slick agent that can handle forwarding metrics and logs to a public API. All of these integrations where great for quick instrumentation of things like Kafka, Cassandra, AWS EC2’s, PostgreSQL etc. . In my use case I needed to create a custom check. Thanks to DataDog’s great documentation I was able to stumble through rather quickly and build my own custom check using a python script that forwarded custom metrics to my DataDog cloud account. Once in the cloud account I was able to trend the metrics.

A custom check is great for a unique application , but if its something others can benefit from like a generic application then developers can build a DataDog Custom Agent-Integration. The integration is a bit more polished than a local custom check and requires a few more hoops to jump through. Its always great to just check the public DataDog Integration GitHub Repo to see if your integration doesn’t already exist.

The next level is a Marketplace Integration. It is just a custom integration that is committed to the private repo called “DatDog/Marketplace”. (more on deploying this later). DataDog Technology Partners are given access to these repos to develop their offering. Some extra steps involved are figuring out the pricing and licensing agreements.

Process Steps

1. Connected with DataDog Sales and pitched my integration idea. They rescheduled a meeting to discuss the Marketplace.
2. Connected with DataDog Technology Partner program team , pitched my integration idea and got approval to start as a Technology Partner.
3. Received confirmation emails and had to login into the Technology Partner portal with my account details. I then followed the steps to request a sandbox account.
4. Received an email with confirmation that my account was ready to use and then logged into www.datadog.hq to access the console for our sandbox account.
5. I then created a Github account linked to a new Github Organization account so that my team and I could use it to develop our integration. We sent the development accounts to our development team contact. After a couple days we had access to the ‘DataDog/marketplace’ repo.
6. Cloned the ‘datadog/marketplace’ repo
7. Read through the developer guide for integration setup. It was a bit confusing at first but the main take away is that the ddev cli utility has built in functions to guide you through the setup of a new integration and validate it as much as possible. The integration can be setup completely in a docker container and tested. The tooling isn’t that bad but it does require some careful reading.
8. Created a PR request on the ‘datadog/marketplace’ that tried to merge the the new folder we built with the integration in it.
9. We worked back and forth on all of the nuances we got wrong. I spent a lot of time copying from other completed integrations and leveraging some ideas as what the format should be.
10. We created vectorized logos, screenshots, and supporting documentation.
11. A DataDog dashboard was created on the console that had all the widgets to present the metrics. Then a JSON file of the dashboard was exported (pretty printed) and stored with the integration.
12. A legal document EULA outlined the terms of usage for the integration.
13. I was surprised how thorough the code review was. They provided feedback from python syntax to layout of the readme.md.

In Review

So far the process has been smooth. I admire the extensive documentation and the focus on developer support. I sometimes have to figure out the nuance details on my own like any ‘real programmer’ would but I do feel like any major roadblocks could be solved by either reaching out to the team or diving into existing code from other integrators.

I have seen DataDog re-invest in new features over the years and its marketplace seems like its going to be very well supported. DataDogs development framework lends itself to great external collaboration. I predict that they will have many more offerings and continue to perfect the developer support.

I asked many founders and integrators how they felt about marketplace offerings. The overwhelming response has been “Its better to control your own destiny” . For most people this is the right approach. I do see a benefit to partnering with DataDog’s marketplace to reach a growing market of DevOps and SecOps professionals and build a proof of concept that can be distributed rapidly. I may one day develop a more performant product outside of the DataDog Integration. But I would still want to provide inter-operability to this growing platform.