DD Instrumentation for Golang x React Application (for local Linux / MacOS)

This guide must be read in conjunction with the official documentation on TrueWatch / Guance.

Setup covered: Datakit, Traces, Logs, Go Profiling, RUM.

Backend stack for demo purposes:

• Language: Golang
• Http Framework: Gin
• ORM: SQLC with database/sql driver
• DB: Postgres

Frontend stack for demo purposes:

• Language: Typescript
• Framework: React + Vite

Part A: Set Up Datakit

1. Follow the cli setup instructions on TrueWatch Console > Integrations.

• If you’re using macOS, it’s recommended to install Orbstack and run your setup inside a Linux VM to avoid errors with the cpu and diskio modules.

2. The default Datakit config may be found and edited:

sudo vi /usr/local/datakit/conf.d/datakit.conf

3. Start Datakit:

sudo datakit service -S

4. Check that the default enabled modules (e.g. cpu, disk, diskio etc.) are all up and running. Try

• datakit check --config
• datakit monitor -V - Check that the inputs are all alive and that there are no errors.
• curl localhost:9529 - Datakit exposes this endpoint to accept telemetry data from your application.

5. Other useful Datakit debug tooling:

• sudo tail -f /var/log/datakit/gin.log - View real-time telemetry POSTs received by Datakit across its exposed endpoints.
• sudo tail -f /var/log/datakit/log | grep ERROR - Displays errors with the enabled modules, if any.

Enabling new modules in Datakit

1. During the instrumentation of your application, you’ll need to enable specific Datakit modules. A complete list of available modules can be found here:: cd /usr/local/datakit/conf.d/samples

2. To enable a module, ensure that you are in the samples subdirectory before running:

sudo cp <module_name>.conf.sample <module_name>.conf

Datakit will automatically enable modules that do not have the .sample suffix.

3. Check the documentation for the relevant module (e.g. ddtrace) and update its configuration in the <module_name>.conf file as needed.

4. Whenever you modify Datakit’s configuration, restart Datakit to apply the changes::

sudo datakit service -R

5. Verify that the newly activated module is running with positive statuses: The Status should show as 'OK' with a valid BodySize.

Part B: [Backend] Enabling APM Traces

This section guides you through configuring nested dd traces that connect Http requests to the corresponding database queries.

Pre-requisites

When configuring nested Datadog tracing, it’s important to verify that your database driver supports trace propagation.

Incompatible drivers may prevent trace context from being passed correctly, causing your Http request and database query to appear as separate traces (different trace IDs) rather than a single unified trace.

1. Add the following code to your main.go:

package main

import (
    sqltrace "github.com/DataDog/dd-trace-go/contrib/database/sql/v2"
	gintrace "github.com/DataDog/dd-trace-go/contrib/gin-gonic/gin/v2"
    "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
)

func main() {
    // setup the dd tracer (ensure that it points to the right Datakit address)
    _ = tracer.Start(tracer.WithAgentAddr("localhost:9529"), tracer.WithService("my-app"), tracer.WithEnv("dev"))
	defer tracer.Stop()

    // trace db queries
    sqltrace.Register("postgres", &pq.Driver{}, sqltrace.WithDBMPropagation(tracer.DBMPropagationModeFull), sqltrace.WithService("my-app-db"))
	pool, err := sqltrace.Open("postgres", "your-db-url")
	if err != nil {
		log.Fatal()
	}
	defer pool.Close()
    queries := sqlc.New(pool)

    // trace http requests 
    r := gin.Default()
	r.Use(gintrace.Middleware("my-app"))

    // create an endpoint to test the tracing
    r.GET("test", func(c *gin.Context) {
        t, _ := tracer.SpanFromContext(c.Request.Context())
        fmt.Println("trace_id : ", t.Context().TraceID()) 
        fmt.Println("span_id : ", t.Context().SpanID())   

		items, err := queries.ListItems(c.Request.Context()) // test a database query of your choosing
		if err != nil {
			c.JSON(http.StatusInternalServerError, err.Error())
		}
		c.JSON(http.StatusOK, items)
	})

    // start the http server
    server := &http.Server{
		Addr:    ":9000",
		Handler: r,
	}
    if err := server.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
		log.Fatal()
	}
}

2. Enable the ddtrace module in Datakit and restart Datakit service.

3. Make a few requests to your app's test endpoint and verify that you can see the request and the corresponding database query in the same Flame chart.

Part C: [Backend] Enabling Logs Collection

The following method outlined collects logs directly from a disk file. For containerized applications, you can opt for one of the other methods outlined here.

1. Ensure that your application write logs to a file on disk.

package main 

import (
	"github.com/rs/zerolog"
	"github.com/rs/zerolog/log"
)

func setupLogger(file *os.File) {
	multi := io.MultiWriter(os.Stdout, file) // writes to both disk file and stdout, modify this as needed
	log.Logger = zerolog.New(multi).With().Timestamp().Caller().Logger()
	zerolog.ErrorStackMarshaler = func(err error) interface{} {
		return string(debug.Stack())
	}
	log.Info().Msg("Setup zerolog.")
}

func main() {
	file, err := os.OpenFile("/<path-to-log-file>/app.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
	if err != nil {
		panic(err)
	}
	defer file.Close()
	setupLogger(file)

	// rest of your code...
}

2. Enable the logging module and restart Datakit. Remember to specify the logfile that Datakit should read from.

3. Add some logs to your application, run your application and verify that you are receiving logs on the cloud console.

Part D: [Backend] Enabling Golang Profiling

1. Follow the documentation here.

• Feel free to use the upgraded Datadog package instead github.com/DataDog/dd-trace-go/v2/profiler.

2. Enable the profiling module, restart Datakit, and verify that the data appears on the cloud console.

Part E: [Frontend] Enabling RUM

1. Follow the comprehensive documentation (here)[https://docs.truewatch.com/real-user-monitoring/#data-sources].