Overview¶

What is Antenna?¶

Antenna is the name of the collector for the Mozilla crash ingestion pipeline. The processor, scheduled task runner, and webapp portions of the crash ingestion pipeline are in Socorro.

For more information about the crash ingestion pipeline and what the collector does, see the Socorro Overview.

Antenna is the collector of the crash ingestion pipeline. It handles incoming crash reports posted by crash reporter clients, generates a crash id which is returned to the client, saves the information, and publishes a crash ids for processing.

Requirements ¶

Antenna is built with the following requirements:

Minimal dependencies

Every dependency we add is another software cycle we have to track causing us to have to update our code when they change.
Make setting it up straight-forward

Antenna should be straight-forward to set up. Minimal configuration options. Good defaults. Good documentation.
Easy to test

Antenna should be built in such a way that it’s easy to write tests for. Tests that are easy to read and easy to write are easy to verify and this will make it likely that the software is higher quality.

High-level architecture ¶

Antenna is the collector of the crash ingestion pipeline.

Data flow ¶

This is the rough data flow:

Crash reporter client submits a crash report via HTTP POST with a multipart/form-data encoded payload.

See Specification: Submitting Crash Reports for details on format.
Antenna’s BreakpadSubmitterResource handles the HTTP POST request.

If the payload is compressed, Antenna uncompresses it.

Antenna extracts the payload.

Antenna throttles the crash report using a ruleset defined in the throttler.

If the throttler rejects the crash, collection ends here.

If the throttler accepts the crash, Antenna generates a crash id.
Then BreakpadSubmitterResource passes the data to the crashmover to save and publish.

If crashstorage is GcsCrashStorage, then the crashmover saves the crash report data to Google Cloud Storage.

If the save is successful, then the crashmover publishes the crash report id to the Google Cloud Pub/Sub standard queue topic for processing.

At this point, the HTTP POST has been handled, the crash id is sent to the crash reporter client and the HTTP connection ends.

Diagnostics ¶

Collector-added fields ¶

Antenna adds several fields to the raw crash capturing information about collection:

metadata

Holds additional properties of the crash report including how it was structured and whether there were any problems with it.

collector_notes: Notes covering what happened during collection. This includes which fields were removed from the raw crash.
dump_checksums: Map of dump name (e.g. upload_file_minidump) to md5 checksum for that dump.
payload: Specifies how the crash annotations were in the crash report. multipart means the crash annotations were encoded in multipart/form-data fields and json means the crash annotations were in a JSON-encoded value in a field named extra.
payload_compressed: 1 if the payload was compressed and 0 if it wasn’t.

submitted_timestamp

The timestamp for when this crash report was collected in UTC in YYYY-MM-DDTHH:MM:SS.SSSSSS format.

uuid

The crash id generated for this crash report.

version

The raw crash schema version. Currently, this is 2.

Logs to stdout ¶

In a production environment, Antenna logs to stdout in mozlog format.

You can see crashes being accepted and saved:

{"Timestamp": 1493998643710555648, "Type": "antenna.breakpad_resource", "Logger": "antenna", "Hostname": "ebf44d051438", "EnvVersion": "2.0", "Severity": 6, "Pid": 15, "Fields": {"host_id": "ebf44d051438", "message": "8e01b4e0-f38f-4b16-bc5a-043971170505: matched by is_firefox_desktop; returned DEFER"}}
{"Timestamp": 1493998645733482752, "Type": "antenna.breakpad_resource", "Logger": "antenna", "Hostname": "ebf44d051438", "EnvVersion": "2.0", "Severity": 6, "Pid": 15, "Fields": {"host_id": "ebf44d051438", "message": "8e01b4e0-f38f-4b16-bc5a-043971170505 saved"}}

Statsd ¶

Antenna sends data to statsd. Read the code for what’s available where and what it means.

Here are some good ones:

breakpad_resource.incoming_crash

Counter. Denotes an incoming crash.
throttle.*

Counters. Throttle results. Possibilities: accept, defer, reject.
breakpad_resource.save_crash.count

Counter. Denotes a crash has been successfully saved.
breakpad_resource.save_queue_size

Gauge. Tells you how many things are sitting in the crashmover_save_queue.

Note

If this number is > 0, it means that Antenna is having difficulties keeping up with incoming crashes.
breakpad_resource.on_post.time

Timing. This is the time it took to handle the HTTP POST request.
breakpad_resource.crash_save.time

Timing. This is the time it took to save the crash to Google Cloud Storage.
breakpad_resource.crash_handling.time

Timing. This is the total time the crash was in Antenna-land from receiving the crash to saving it to Google Cloud Storage.

Sentry ¶

Antenna works with Sentry and will send unhandled startup errors and other unhandled errors to Sentry where you can more easily see what’s going on. You can use the hosted Sentry or run your own Sentry instance–either will work fine.

Cloud storage file hierarchy ¶

If you use the Google Cloud Storage crashstorage component, then crashes get saved in this hierarchy in the bucket:

/v1/raw_crash/<DATE>/<CRASHID>
/v1/dump_names/<CRASHID>

And then one or more dumps in directories by dump name:

/v1/<DUMP_NAME>/<CRASHID>

Note that upload_file_minidump gets converted to dump.

For example, a crash with id 00007bd0-2d1c-4865-af09-80bc00170413 and two dumps “upload_file_minidump” and “upload_file_minidump_flash1” gets these files saved:

v1/raw_crash/20170413/00007bd0-2d1c-4865-af09-80bc00170413

    Raw crash in serialized in JSON.

v1/dump_names/00007bd0-2d1c-4865-af09-80bc00170413

    Map of dump_name to file name serialized in JSON.

v1/dump/00007bd0-2d1c-4865-af09-80bc00170413

    upload_file_minidump dump.

v1/upload_file_minidump_flash1/00007bd0-2d1c-4865-af09-80bc00170413

    upload_file_minidump_flash1 dump.