BSky Experiments

This repo has contains some fun Go experiments interacting with BlueSky via the AT Protocol.

The Makefile contains some useful targets for building and running the experiments.

Graph Builder (for the BSky Atlas)

The GraphBuilder is the main project in this repo, it constructs a graph of interactions between users on the platform based on all commits/posts that come in from the firehose.

The Graph is stored in a shared memory datastructure that's dumped to disk in a binary format every 30 seconds, overwriting its past checkpoint.

Every 30 minutes, it starts writing to a new file with an appropriate timestamp so you can track change in the graph over time.

Graph data is written to data/social-graph.bin for the latest version and data/social-graph-{YYYY}_{MM}_{DD}_{HH}_{MM}.bin for the checkpoints using the date and time according to current time UTC.

Running the Graph Builder

1. Copy the contents of .env.example to .env in the root project directory.

2. Add your BSky App Password credentials (using your normal BSky email address as the username), replacing the placeholder in ATP_AUTH=.

3. For OTLP Tracing, configure the OTEL_EXPORTER_OTLP_ENDPOINT= to point at a valid OTLPHTTP trace collector.

   • If you don't want this functionality, delete the variable from the .env file and it will disable the feature.

4. Logging is by default handled by a Loki agent running in my local network, you should delete the following block in the build/graph-builder/docker-compose.yml file:

   logging:
     driver: loki
     options:
     loki-url: "http://10.0.6.7:3100/loki/api/v1/push"

5. If you'd like to collect the text and references in EVERY post on BSky, update REGISTRY_DB_CONNECTION_STRING= in .env to be a valid Postgres connection string to an accessible database.

   • If you don't want this functionality, delete the variable from the .env file and it will disable the feature.

To build containers and start up the Graph Builder, run:

$ make graph-builder-up

The only dependency required for this is Docker.

A docker compose version of the setup preconfigured with redis and postgres can be ran with the following command:

$ docker compose -f docker-compose.yml up

the docker compose setup uses a few additional .env file variables:

• REDIS_ADDRESS is used for the redis instance and is set in the default setup with REDIS_ADDRESS=redis:6379 `
• POSTGRES_DB is used for the database name and should match the end of the REGISTRY_DB_CONNECTION_STRING, the default sets POSTGRES_DB=registry
• POSTGRES_USER is used for the database user and should match the health check command in the docker-compose file ie. pg_isready -U $POSTGRES_USER, the defaults sets POSTGRES_USER=postgres
• POSTGRES_PASSWORD is used for the database user's password, the default setup uses POSTGRES_PASSWORD=password

Metrics

Metrics and debug routes exist at {host}:6060/metrics and {host}:6060/debug/pprof/{profile} for Prometheus metrics on data consumed and pprof-based memory and CPU profiling of the graph-builder.