I think all mentioned issues are related to the implementation of the user probes, not the pattern.
You mentioned an observation methods, but essentially they are absolutely the same as hooks, just inverted (with a bit less overhead on branching and hook call). E.g. your example with bytesReceived counter can be implemented with atomic operation and further export on demand by some other goroutine.
The thing I keep in mind is that state is mutated far more than it is observed. You might handle 1000 or more requests per second and only export the number of requests once per second or less. In light of this ratio it’s important to make the recording path as simple and cheap as possible, and it’s ok if the observing path has to be complicated to compensate.
I usually refuse to use atomic increments for services because it scales very poorly. Even a mutex-protected increment scales much better than atomic increment.
But it doesn't work well in Go by several reasons.
For example, you usually define interface not in the package where its implemented, but where its needed. Thus you must provide such wrappers in every place where some (subset) of the `Client`'s methods are used.
Another reason is that such thing rejects an ability to use your struct as is, without interfaces (and thus sometimes without heap allocation for your struct).
And, finally, if you want to adopt such approach in proprietary/internal software, where interfaces usually change more often than in libraries, you will change code in N places instead of 1 (in best case); where N is number of instrumentation methods you want to provide "as feature".
We haven't modified the compiler but instead plugged into it an external instrumentation tool using the compiler option `-toolexec`.
With this option, the compiler invokes every toolchain binary (compile, asm, link, etc.) through the provided program. So you can basically write a proxy program intercepting calls to `compile` to do source-code instrumentation, exactly like you would with go generate, but now automatically done during compilation on every package.
As a library developer, don't you need to define exactly the same trace points (hooks) in your code?
Am I understand it right, that you suggest instead of doing it in a generic way (any tracing method could be plugged in this way), to stick to one particular library?
Yes, because then the library developer can define all of the things that COULD be interesting, and leave it to the user of the library to decide WHAT to record (filters), and WHERE to record it (exporters).
And since this is a standard, it means that when a user includes a bunch of disparate libraries/remote calls in his app, he can define his instrumentation wishes once for everything during app initialization. The libraries can even cooperate since they use the same underlying API and concepts for instrumentation.
> Yes, because then the library developer can define all of the things that COULD be interesting, and leave it to the user of the library to decide WHAT to record (filters), and WHERE to record it (exporters).
I agree that this uppercase words can be called as somewhat purposes of the tracing API. I believe also that I was talking exactly about the same approach in the article.
But I can't get how sticking to one particular library will help here? In other words, I think it will just greatly reduce the usage options.
What Im saying is that you can leave the generic hooks in the library (as it described in the article and gtrace) and then let the user choose which library to use inside what hooks.
With a standard library and API that everyone adheres to, you just pass in some initial configuration during app initialization to decide which libraries record what kinds of information. The user only touches one file to enable and choose instrumentation.
With hooks, you get nothing unless you hook, which means you have to write hooks for everything you're interested in (and also come up with a way to export the data), and so the user must touch code all over the place.
Tracing term is kind of overloaded. In this article its used not as _distributed tracing_, but as _instrumenting_. E.g., Linux's trace points, or any other tools such dtrace, strace and so on. So it doesn't relate to systems design decisions.
You mentioned an observation methods, but essentially they are absolutely the same as hooks, just inverted (with a bit less overhead on branching and hook call). E.g. your example with bytesReceived counter can be implemented with atomic operation and further export on demand by some other goroutine.