Configuration¶
Introduction¶
Antenna uses environment configuration to define its behavior.
The local development environment is configured in the my.env
and
docker/config/local_dev.env
env files and that configuration is pulled in
when you run Antenna using docker compose
.
In a server environment, configuration is pulled in from the process environment.
Here’s an example. This uses statsd with datadog extensions installed on localhost for metrics.
# Metrics things
STATSD_NAMESPACE=mcboatface
# BreakdpadSubmitterResource settings
CRASHMOVER_CRASHSTORAGE_CLASS=antenna.ext.gcs.crashstorage.GcsCrashStorage
# GcsCrashStorage settings
CRASHMOVER_CRASHSTORAGE_BUCKET_NAME=org-myorg-mybucket
Gunicorn configuration¶
For Gunicorn configuration, see Dockerfile
. You’ll want to set the
following:
- GUNICORN_WORKERS¶
- Parser:
str
- Default:
“1”
- Required:
No
The number of Antenna processes to spin off. We use 2x+1 where x is the number of processors on the machine we’re using.
This is the
workers
Gunicorn configuration setting.https://docs.gunicorn.org/en/stable/settings.html#workers
This is used in bin/run_web.sh.
- GUNICORN_WORKER_CLASS¶
- Parser:
str
- Default:
“sync”
- Required:
No
This is the
worker-class
Gunicorn configuration setting.https://docs.gunicorn.org/en/stable/settings.html#worker-class
This is used in bin/run_web.sh.
- GUNICORN_MAX_REQUESTS¶
- Parser:
str
- Default:
“0”
- Required:
No
If set to 0, this does nothing.
For a value greater than 0, the maximum number of requests for the worker to serve before Gunicorn restarts the worker.
This is the
ma-requests
Gunicorn configuration setting.https://docs.gunicorn.org/en/stable/settings.html#max-requests
This is used in bin/run_web.sh.
- GUNICORN_MAX_REQUESTS_JITTER¶
- Parser:
str
- Default:
“0”
- Required:
No
Maximum jitter to add to
GUNICORN_MAX_REQUESTS
setting.This is the
ma-requests-jitter
Gunicorn configuration setting.https://docs.gunicorn.org/en/stable/settings.html#max-requests-jitter
This is used in bin/run_web.sh.
- CMD_PREFIX¶
- Parser:
str
- Default:
“”
- Required:
No
Specifies a command prefix to run the Gunicorn process in.
This is used in bin/run_web.sh.
Application¶
First, you need to configure the application-scoped variables.
- Configuration¶
These are defaults appropriate for a server environment, so you may not have to configure any of this.
Configuration summary:
Setting
Parser
Required?
str
str
bool
str
int
str
str
Configuration options:
- BASEDIR¶
- Parser:
str
- Default:
“/home/docs/checkouts/readthedocs.org/user_builds/antenna/checkouts/stable”
- Required:
No
The root directory for this application to find and store things.
- LOGGING_LEVEL¶
- Parser:
str
- Default:
“INFO”
- Required:
No
The logging level to use. DEBUG, INFO, WARNING, ERROR or CRITICAL
- LOCAL_DEV_ENV¶
- Parser:
bool
- Default:
“False”
- Required:
No
Whether or not this is a local development environment.
- STATSD_HOST¶
- Parser:
str
- Default:
“localhost”
- Required:
No
Hostname for statsd server.
- STATSD_PORT¶
- Parser:
int
- Default:
“8125”
- Required:
No
Port for statsd server.
- SECRET_SENTRY_DSN¶
- Parser:
str
- Default:
“”
- Required:
No
Sentry DSN to use. See https://docs.sentry.io/quickstart/#configure-the-dsn for details. If this is not set an unhandled exception logging middleware will be used instead.
- HOSTNAME¶
- Parser:
str
- Default:
“build-26134660-project-52135-antenna”
- Required:
No
Identifier for the host that is running Antenna. This identifies this Antenna instance in the logs and makes it easier to correlate Antenna logs with other data. For example, the value could be a an instance id, pod id, or something like that.
If you do not set this, then
socket.gethostname()
is used instead.
Breakpad crash resource¶
- component antenna.breakpad_resource.BreakpadSubmitterResource¶
Handles incoming breakpad-style crash reports.
This handles incoming HTTP POST requests containing breakpad-style crash reports in multipart/form-data format.
It can handle compressed or uncompressed POST payloads.
It parses the payload from the HTTP POST request, runs it through the throttler with the specified rules, generates a crash_id, returns the crash_id to the HTTP client, and passes the crash report data to the crashmover.
Configuration summary:
Setting
Parser
Required?
str
Configuration options:
- BREAKPAD_DUMP_FIELD¶
- Parser:
str
- Default:
“upload_file_minidump”
- Required:
No
The name of the field in the POST data for dumps.
Throttler¶
- component antenna.throttler.Throttler¶
Accept or reject incoming crashes based on specified rule set.
The throttler can throttle incoming crashes using the content of the crash. To throttle, you set up a rule set which is a list of
Rule
instances. That goes in a Python module which is loaded at run time.If you don’t want to throttle anything, use this:
BREAKPAD_THROTTLER_RULES=antenna.throttler.ACCEPT_ALL
If you want to support all products, use this:
BREAKPAD_THROTTLER_PRODUCTS=antenna.throttler.ALL_PRODUCTS
To set up a rule set, put it in a Python file and define the rule set there. For example, you could have file
myruleset.py
with this in it:from antenna.throttler import Rule rules = [ Rule('ProductName', 'Firefox', 100), # ... ]
then set
BREAKPAD_THROTTLER_RULES
to the path for that. For example, depending on the current working directory andPYTHONPATH
, the above could be:BREAKPAD_THROTTLER_RULES=myruleset.rules
Configuration summary:
Setting
Parser
Required?
antenna.throttler.parse_attribute
antenna.throttler.parse_attribute
antenna.throttler.parse_attribute
Configuration options:
- BREAKPAD_THROTTLER_RULES¶
- Parser:
antenna.throttler.parse_attribute
- Default:
“antenna.throttler.MOZILLA_RULES”
- Required:
No
Python dotted path to ruleset
- BREAKPAD_THROTTLER_PRODUCTS¶
- Parser:
antenna.throttler.parse_attribute
- Default:
“antenna.throttler.MOZILLA_PRODUCTS”
- Required:
No
Python dotted path to list of supported products
- BREAKPAD_THROTTLER_PRODUCT_PACKAGENAMES¶
- Parser:
antenna.throttler.parse_attribute
- Default:
“antenna.throttler.MOZILLA_PRODUCT_PACKAGENAMES”
- Required:
No
Python dotted path to map of product -> list of supported packagenames
Crash mover¶
- component antenna.crashmover.CrashMover¶
Handles saving and publishing crash reports.
The crashmover saves the crash using the configured crashstorage class and publishes it using the configured crashpublish class.
Note
From when a crash comes in to when it’s saved by the crashstorage class, the crash is entirely in memory. Keep that in mind when figuring out how to scale your Antenna nodes.
The most important configuration bit here is choosing the crashstorage class.
For example:
CRASHMOVER_CRASHSTORAGE_CLASS=antenna.ext.gcs.crashstorage.GcsCrashStorage
Configuration summary:
Setting
Parser
Required?
everett.manager.parse_class
everett.manager.parse_class
int
int
Configuration options:
- CRASHMOVER_CRASHSTORAGE_CLASS¶
- Parser:
everett.manager.parse_class
- Default:
“antenna.ext.crashstorage_base.NoOpCrashStorage”
- Required:
No
The class in charge of storing crashes.
- CRASHMOVER_CRASHPUBLISH_CLASS¶
- Parser:
everett.manager.parse_class
- Default:
“antenna.ext.crashpublish_base.NoOpCrashPublish”
- Required:
No
The class in charge of publishing crashes.
- CRASHMOVER_MAX_ATTEMPTS¶
- Parser:
int
- Default:
“5”
- Required:
No
Maximum number of attempts to save or publish a crash before giving up.
- CRASHMOVER_RETRY_SLEEP_SECONDS¶
- Parser:
int
- Default:
“2”
- Required:
No
Seconds to sleep between attempts to save or publish a crash.
Crash storage¶
For crash storage, you have three options one of which is a no-op for debugging.
NoOpCrashStorage¶
The NoOpCrashStorage
class is helpful for debugging, but otherwise shouldn’t
be used.
- component antenna.ext.crashstorage_base.NoOpCrashStorage¶
This is a no-op crash storage that logs crashes it would have stored.
It keeps track of the last 10 crashes in
.saved_things
instance attribute with the most recently stored crash at the end of the list. This helps when writing unit tests for Antenna.Configuration summary:
Setting
Parser
Required?
Configuration options:
Filesystem¶
The FSCrashStorage
class will save crash data to disk. If you choose this,
you’ll want to think about what happens to the crash after Antenna has saved it
and implement that.
- component antenna.ext.fs.crashstorage.FSCrashStorage¶
Save raw crash files to the file system.
This generates a tree something like this which mirrors what we do on GCS:
<FS_ROOT>/ <YYYYMMDD>/ raw_crash/ <CRASHID>.json dump_names/ <CRASHID>.json <DUMP_NAME>/ <CRASHID>
Couple of things to note:
This doesn’t ever delete anything from the tree. You should run another process to clean things up.
If you run out of disk space, this component will fail miserably. There’s no way to recover from a full disk–you will lose crashes.
FIXME(willkg): Can we alleviate or reduce the likelihood of the above?
When set as the CrashMover crashstorage class, configuration for this class is in the
CRASHMOVER_CRASHSTORAGE
namespace.Example:
CRASHMOVER_CRASHSTORAGE_FS_ROOT=/tmp/whatever
Configuration summary:
Setting
Parser
Required?
str
Configuration options:
- CRASHMOVER_CRASHSTORAGE_FS_ROOT¶
- Parser:
str
- Default:
“/tmp/antenna_crashes”
- Required:
No
path to where files should be stored
Google Cloud Storage¶
The GcsCrashStorage
class will save crash data to Google Cloud Storage.
- component antenna.ext.gcs.crashstorage.GcsCrashStorage¶
Save raw crash files to GCS.
This will save raw crash files to GCS in a pseudo-tree something like this:
<BUCKET> v1/ dump_names/ <CRASHID> <DUMPNAME>/ <CRASHID> raw_crash/ <YYYYMMDD>/ <CRASHID>
Authentication
The google cloud sdk will automatically detect credentials as described in https://googleapis.dev/python/google-api-core/latest/auth.html:
If you’re running in a Google Virtual Machine Environment (Compute Engine, App Engine, Cloud Run, Cloud Functions), authentication should “just work”.
If you’re developing locally, the easiest way to authenticate is using the Google Cloud SDK:
$ gcloud auth application-default login
If you’re running your application elsewhere, you should download a service account JSON keyfile and point to it using an environment variable:
$ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"
Local emulator
If you set the environment variable
STORAGE_EMULATOR_HOST=http://host:port
, then this will connect to a local GCS emulator.When set as the CrashMover crashstorage class, configuration for this class is in the
CRASHMOVER_CRASHSTORAGE
namespace.Example:
CRASHMOVER_CRASHSTORAGE_BUCKET_NAME=mybucket
Configuration summary:
Setting
Parser
Required?
str
Yes
Configuration options:
- CRASHMOVER_CRASHSTORAGE_BUCKET_NAME¶
- Parser:
str
- Required:
Yes
Google Cloud Storage bucket to save to. Note that the bucket must already have been created.
Crash publish¶
For crash publishing, you have two options one of which is a no-op.
NoOpCrashPublish¶
The NoOpCrashPublish
class is helpful for debugging and also if you don’t
want Antenna to be publishing crash ids somewhere.
- component antenna.ext.crashpublish_base.NoOpCrashPublish¶
No-op crash publish class that logs crashes it would have published.
It keeps track of the last 10 crash ids in
.published_things
instance attribute with the most recently published crash id at the end of the list. This helps when writing unit tests for Antenna.
Google Pub/Sub¶
The PubSubCrashPublish
class will publish crash ids to a Google Pub/Sub
topic.
- component antenna.ext.pubsub.crashpublish.PubSubCrashPublish¶
Publisher to Pub/Sub.
Required GCP things
To use this, you need to create:
Google Compute project
topic in that project
subscription for that topic so you can consume queued items
If something in the above isn’t created, then Antenna may not start.
Verification
This component verifies that it can publish to the topic by publishing a fake crash id of
test
. Downstream consumer should throw this out.Authentication
The google cloud sdk will automatically detect credentials as described in https://googleapis.dev/python/google-api-core/latest/auth.html:
If you’re running in a Google Virtual Machine Environment (Compute Engine, App Engine, Cloud Run, Cloud Functions), authentication should “just work”.
If you’re developing locally, the easiest way to authenticate is using the Google Cloud SDK:
$ gcloud auth application-default login
If you’re running your application elsewhere, you should download a service account JSON keyfile and point to it using an environment variable:
$ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"
Local emulator
If you set the environment variable
PUBSUB_EMULATOR_HOST=host:port
, then this will connect to a local Pub/Sub emulator.When set as the BreakpadSubmitterResource crashpublish class, configuration for this class is in the
CRASHMOVER_CRASHPUBLISH
namespace.You need to set the project id and topic name.
Configuration summary:
Setting
Parser
Required?
str
Yes
str
Yes
float
Configuration options:
- CRASHMOVER_CRASHPUBLISH_PROJECT_ID¶
- Parser:
str
- Required:
Yes
Google Cloud project id.
- CRASHMOVER_CRASHPUBLISH_TOPIC_NAME¶
- Parser:
str
- Required:
Yes
The Pub/Sub topic name to publish to.
- CRASHMOVER_CRASHPUBLISH_TIMEOUT¶
- Parser:
float
- Default:
“5”
- Required:
No
The amount of time in seconds individual rpc calls to pubsub are allowed to take before giving up. This is passed to PublisherOptions. From 2018 through 2023 Mozilla’s internal data platform’s use of pubsub generally saw publish response times less than 150ms, and had one incident where response times were approximately 1 second. Based on that experience this has a default of 5 seconds.