I recently built a concurrent URL health checker in Go.
The idea was straightforward: maintain a list of URLs, distribute them among a fixed number of workers, apply a timeout to every HTTP request, and collect the results through a channel.
Instead of starting one goroutine for every URL, I created a small worker pool with three workers. This gave me control over how many HTTP requests could run at the same time.
On paper, the design looked correct.
The program compiled successfully. It started without a panic. Every HTTP request had a three-second timeout.
But when I ran it, the terminal remained completely silent.
No URL status. No timeout message. Not even the "URL Health Check Started" message.
The program started and then nothing arrived — not even the line printed before the result loop.
The program had deadlocked before it could print a single result.
The original worker-pool setup that compiled successfully but never reached the result collection loop.
The first clue: the timeout was not helping
Each worker created a context with a three-second timeout before making an HTTP request.
That meant a slow or unreachable website should not block a worker forever. After three seconds, the request should either complete or return a timeout error.
Still, the program remained stuck for much longer than three seconds.
This was an important clue: the HTTP request itself was not the main problem. The workers were getting stuck after completing their requests.
Every HTTP request had a timeout, but workers could still block while trying to send their results.
Understanding where the program stopped
Both jobs and results were unbuffered channels.
An unbuffered channel does not store a value. A send can complete only when another goroutine is ready to receive that value.
The main goroutine started three workers and then began sending URLs into the jobs channel.
The first three URLs could be received by the three available workers.
After that, the main goroutine tried to send another URL.
Meanwhile, the workers completed their first HTTP requests and attempted to send their results into the results channel.
That created the deadlock:
- The main goroutine was blocked while trying to submit another job.
- Every worker was blocked while trying to publish a result.
- The result receiver had not started yet because it appeared after the job-submission loop.
The program could not make any further progress.
The job loop and close(jobs) run inline on the main goroutine — nothing here is receiving a result.
The effective flow looked like this:
Main → jobs → workers → results → mainThe cycle itself was not the problem. The problem was that the main goroutine was responsible for both submitting jobs and receiving results, but it tried to finish the first responsibility before starting the second one.
Because workers could not send their results, they could not return to the jobs channel to receive more work.
Confirming the deadlock with a goroutine dump
Instead of relying only on reading the code, I inspected the state of the running goroutines.
I built and ran the program, waited for it to freeze, and then generated a goroutine dump using Ctrl+\ on macOS.
The dump showed:
- Goroutine 1 blocked on a channel send at line 115.
- All three workers blocked on channel sends at line 39.
That matched the suspected deadlock exactly.
The main goroutine was trying to send the next job, while every worker was trying to send a result.
The goroutine dump confirmed that main was blocked while submitting a job and every worker was blocked while sending a result.
This became the most useful part of the debugging process. The program did not report a normal deadlock panic because HTTP and runtime goroutines were still present. The goroutine dump showed the actual blocking locations.
Fixing the pipeline
The job producer needed to run independently from the result consumer.
I moved the loop that submits URLs into its own goroutine. That allowed the main goroutine to start receiving results immediately.
Now the pipeline could make progress in both directions:
Producer → jobs → workers → results → consumerIf all workers were temporarily blocked while sending results, the main goroutine could receive those results. Once a result was received, that worker could return to the jobs channel and accept another URL.
The goroutine responsible for producing jobs also became the owner of the jobs channel. After submitting the final URL, it closed that channel.
This ownership rule made the design easier to reason about:
- The producer closes
jobs. - Workers only receive from
jobs. - Workers only send to
results. - The result-coordination goroutine closes
results. - Main only receives from
results.
Moving job submission into a separate goroutine allowed jobs and results to flow concurrently.
Fixing the second blocking bug
After finding the channel deadlock, I noticed another problem.
The program incremented the WaitGroup counter once for every worker. However, the worker function never called wg.Done().
That meant the goroutine waiting on wg.Wait() could never finish, even after all jobs had been processed.
As a result, the results channel would never be closed, and the final result loop would wait forever.
The worker needed to call Done exactly once when it exited. I added defer wg.Done() at the beginning of the worker function.
Using defer here was helpful because the completion signal would run when the worker returned, regardless of how the loop finished.
Every WaitGroup.Add(1) must eventually have a corresponding Done().
A third bug appeared after the program started working
Once the pipeline was fixed, the checker could finally print results.
However, the URL list contained endpoints that deliberately returned HTTP 400, 404, 500, 502 and 503 responses.
The checker was reporting all of them as UP.
The reason was that the worker only checked the error returned by the HTTP client.
In Go, receiving HTTP 500 does not normally produce an error from http.Client.Do. The HTTP request was successfully sent, and the server successfully returned an HTTP response. That response simply contained an unsuccessful status code.
There are two different kinds of failure here:
- A transport failure, such as a timeout, DNS failure or connection error.
- An HTTP-level failure, such as 404 or 500.
My original implementation handled the first case but ignored the second.
The worker therefore needed to inspect resp.StatusCode and apply an explicit health policy.
For this checker, I decided that successful status codes should be reported as healthy, while error status codes should be reported as unhealthy.
The list deliberately includes endpoints that answer with 400, 404, 500, 502 and 503.
An HTTP response is not automatically a healthy response — status codes need their own classification.
The final result
After correcting the channel flow, worker completion tracking and status-code handling, the checker could process URLs with a fixed number of workers and print results as they arrived.
The order of the output was not the same as the order of the input URLs. That was expected because faster HTTP requests completed before slower ones.
This was also a useful change in how I thought about concurrent programs: preserving input order and processing concurrently are separate design decisions.
The corrected checker streamed results and exited after all three workers completed.
What I learned
The biggest lesson was that using goroutines and channels does not automatically make a program correctly concurrent.
The communication flow still has to be designed carefully.
From this project, I learned to ask:
- Can a sender reach a receiver at the same time?
- Who owns and closes each channel?
- Is the WaitGroup tracking workers or individual jobs?
- Can the result consumer start before every job has been submitted?
- Does an HTTP response actually mean the target is healthy?
- What evidence can confirm where goroutines are blocked?
I also learned that timeouts only protect the operation using the timeout. My HTTP timeout could stop a slow request, but it could not prevent the worker from blocking forever on a channel send.
The final bug was not one mistake but a chain of independent mistakes:
- The producer blocked before the consumer started.
- Workers never marked themselves complete.
- HTTP error responses were classified as healthy.
That is what made this small project useful. It moved me beyond simply starting goroutines and helped me understand how goroutine lifecycles, channel ownership and backpressure work together.
A program can compile successfully and still have no possible path forward.