Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

☠️ WIP ☠️ Receive Traces from a port instead of erldist #154

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

andytill
Copy link
Owner

@andytill andytill commented Aug 27, 2017

This is the next step in making erlyberly safe to use in production.

Currently, erlyberly receives trace messages to a process which then forwards them to the erlyberly node via erlang distribution.

With this change, trace messages will be sent via a normal TCP socket without erlang distribution which is much lower overhead. On the remote node, a TCP server socket is opened when erlyberly connects as an erlang node. Erlyberly connects to the server socket and will begin receiving trace messages.

Benchmarks showing the performance improvements for tracing to ports are here: https://github.com/andytill/traceperf

The existing method of preventing overload by checking the message queue length of the process receiving the trace messages is no longer possible because the traces no longer exist as erlang messages and are sent directly to the port. Instead, the TCP tracer sockets supports load shedding.

Now that the trace records are sent directly to erlyberly over TCP rather than through a "middle man" process, there is no opportunity to analyse them on the remote node. The following functionality has been changed:

Code Reloading

Traces are now checked to see if they are a code load or reload on the erlyberly side when the trace is received. If TraceManager detects a code reload then it issues an rpc to get refreshed MFAs for the module, which may have been changed.

Showing the Registered Process

The registered name of the process is not given as part of the trace log, it only contains the process pid. Erlyberly now maintains its own registry of process names by initially getting all pids to names and putting its own trace on erlang:register/2 and erlang:unregister/1.

  • Need to flush the cache of pid to names because it may accrue dead processes.
Stack Traces

An rpc is made to the remote node to decode the stack trace binary when it needs to be viewed as part of the trace viewe by double clicking the trace.

Highlighted Record Field Names in Traces

Records would be highlighted if they were in one of the OTP behaviour handle callbacks. It is not possible to do this in the same way, a more general approach to highlight records in any function call is possible by if erlyberly kept a registry of function signatures and what they contained by analysing the AST for a module that was trace but it is not part of this PR. For now this feature is removed.

Highlighting records in process state is not affected.

Remaining Tasks

  • Handle the drop message <<1, Num_dropped_messages:32>>
  • Overload testing, is the load shedding in the TCP tracer port sufficient or does there need to be a time limit feature?

@andytill andytill changed the title Trace to erlang ports Trace to Erlang Ports (Work in Progress) Sep 9, 2017
@andytill andytill changed the title Trace to Erlang Ports (Work in Progress) ☠️ WIP ☠️ Receive Traces from a port instead of erldist Oct 19, 2017
Andy Till and others added 8 commits October 20, 2017 00:16
…gainst an object to string instead of actual trace data.
…his was broken because we receive the stack trace as a binary and there is no middle man process to decode it. Now, we store the binary and then send it back to the remote node via rpc to be decoded when we need to view it, and cache that.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant