>_
GolangStepByStep
Software Engineer

HTTP Clients

Timeouts, retries, connection pooling, transport tuning

# What is an HTTP Client? (The Suppy Truck Analogy)

If an HTTP Server is the kitchen taking orders, the HTTP Client is the restaurant's supply truck.

Sometimes, your kitchen needs ingredients from the outside world (like Stripe for payments, AWS for file storage, or Twitter for OAuth). To get these, your Go application must leave its own building, drive over the internet (TCP/IP), knock on another server's door, and ask for data.

The tool used to construct the truck, format the request, wait for the response, and haul the data back is called an HTTP Client.

# Level 1: The One-Liner (Beginner)

Go provides extremely simple one-liners to fetch data. Making a GET request is shockingly easy:

package main

import (
    "fmt"
    "io"
    "net/http"
)

func main() {
    // 1. Send the truck out to fetch Google's homepage
    resp, err := http.Get("https://www.google.com")
    if err != nil {
        panic(err)
    }
    
    // 2. CRUCIAL RULE: You MUST close the response body!
    // Defer ensures this safely happens when the function exits.
    defer resp.Body.Close()

    // 3. Read the payload the server gave us
    body, _ := io.ReadAll(resp.Body)
    fmt.Printf("Received %d bytes from Google!", len(body))
}

While `http.Get` is fun for scripts, it is highly dangerous in real-world servers because it uses Go's "Default Client", which has zero timeouts. If Google's servers froze, your program would simply hang forever waiting.

# Level 2: Real Clients Have Timeouts (Intermediate)

To survive in the real world, your supply truck needs a clock. If the third-party server does not answer in 5 seconds, your truck needs to give up, go home, and report an error, otherwise your own customers will be blocked.

package main

import (
    "fmt"
    "net/http"
    "time"
)

func main() {
    // 1. Build a Custom Client with a strict Time Limit
    client := &http.Client{
        Timeout: 5 * time.Second, // It will literally execute the truck driver if it takes longer than 5s
    }

    // 2. Use YOUR client, not the Default global package!
    resp, err := client.Get("https://api.github.com")
    if err != nil {
        fmt.Println("API failed or timed out:", err)
        return
    }
    defer resp.Body.Close()

    // 3. CAUTION: A 404 or 500 error does NOT trigger the 'err' above!
    // The connection succeeded, the API just told us off. We must check the StatusCode:
    if resp.StatusCode >= 400 {
        fmt.Printf("Uh oh! The server replied with error code: %d\n", resp.StatusCode)
        return
    }
    
    fmt.Println("Success! Status 200 OK.")
}

# Level 3: Adding Headers & JSON (Advanced)

If you need to access a protected API (like Stripe), you can't just use `.Get()`. You have to present an ID Badge (An `Authorization Header`) and speak their language (`Content-Type: application/json`).

To do this, we manually build an *http.Request object from scratch, modify it, and then hand it to our client's client.Do() method.

func FetchSecureData() {
    client := &http.Client{Timeout: 10 * time.Second}
    
    // 1. Build the Request manually! (Method, URL, Body)
    req, err := http.NewRequest("GET", "https://api.stripe.com/v1/customers", nil)
    if err != nil {
        panic(err)
    }
    
    // 2. Inject Custom Headers
    req.Header.Add("Authorization", "Bearer sk_test_123456789")
    req.Header.Add("Accept", "application/json")
    
    // 3. Execute the Request using 'Do'
    resp, err := client.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    
    fmt.Println("Request executed!")
}

Tip: http.NewRequestWithContext(ctx, "GET", url, nil) is the absolute best-practice way to build requests, as it allows your requests to be cancelled mid-flight if the user aborts!

# Level 4: Transports & Connection Pools (Expert)

Every time you do a GET request over HTTPS, a complex "handshake" happens with the remote server. Doing this handshake a thousand times a second is incredibly CPU-heavy.

Because of this, Go's Client has a hidden engine called the http.Transport. By default, it automatically saves recent connections (Keep-Alive TCP pools). But for high-throughput microservices, you often need to tune this engine so the pool holds more connections than the default!

// A Hyper-Optimized HTTP Client for high-traffic microservices!
func SetupExpertClient() *http.Client {
    customTransport := &http.Transport{
        // Max concurrent connections across all hosts
        MaxIdleConns:        100,  
        
        // Essential for microservices: How many pooled connections per specific Domain name (Default is a measly 2!)
        MaxIdleConnsPerHost: 100,  
        
        // Kick them out of the pool if inactive for 90s
        IdleConnTimeout:     90 * time.Second,
        
        // Timeout just for the TCP Handshake phase
        TLSHandshakeTimeout: 10 * time.Second, 
    }

    return &http.Client{
        Transport: customTransport,
        Timeout:   30 * time.Second,
    }
}

Tuning MaxIdleConnsPerHost from the default (2) to 100 is often the secret sauce that stops high-scale applications from crashing due to port exhaustion (running out of TCP ports)!

practice & review