Aiven's Free Kafka Plan

This is the one managed Kafka 129 service in this chapter that ran for real, driven through Aiven's REST API (https://api.aiven.io/v1 67,005 ) by small Python scripts in demos/ch06/aiven/ that read a personal token from a file and redact the account's e-mail address and project name (<project>) from all they print. The free plan is free-0. create.py refuses to continue unless the plan's price is zero in every region it is offered, then creates the service and waits for it:

aiven/create.py (excerpt): check that the plan is free, then create the servicePython

_, t = call("GET", f"/project/{PROJECT}/service_types")
plan = next(p for p in t["service_types"]["kafka"]["service_plans"] if p["service_plan"] == PLAN)
prices = {r["price_usd"] for r in plan["regions"].values()}
...
if prices != {"0.000"} or "free" not in PLAN:
    sys.exit("not a free plan: stopping")
...
st, r = call("POST", f"/project/{PROJECT}/service",
             {"service_name": SERVICE, "service_type": "kafka", "plan": PLAN, "cloud": CLOUD})

Output

plan free-0: price_usd per hour ['0.000'] in 9 clouds; do-fra: 2 CPU, 2048 MB RAM, 48 GB disk,
  2 nodes
HTTP 200: booknest-kafka REBUILDING plan=free-0 cloud=do-fra
RUNNING after 336 s; Kafka 4.3 nodes 2
  kafka endpoint: booknest-kafka-<project>.g.aivencloud.com 12319 certificate dynamic primary

Five and a half minutes later BookNest had a two-broker Kafka 4.3 cluster on DigitalOcean 553 in Frankfurt, one of the free plan's nine regions. Its only listener authenticated clients by certificate (mutual TLS, mTLS and OAuth). Three more API calls (setup.py, sasl.py, users.py) created the topic booknest.order-events with two partitions and two replicas, switched on SASL/SCRAM beside certificates (a second listener appeared on port 12330), and added a user booknest-analytics with an Aiven ACL allowing it read on that topic only. The clients need nothing Aiven-specific, only connection settings:

aiven/aiven_clients.py (excerpt): the two ways in, from files the setup scripts savedPython

D = pathlib.Path("/home/dev/v7-l3/ch06/aiven-secrets")
CERT = {"bootstrap.servers": (D / "bootstrap").read_text(),      # client certificate (mTLS)
        "security.protocol": "SSL", "ssl.ca.location": str(D / "ca.pem"),
        "ssl.certificate.location": str(D / "service.cert"),
        "ssl.key.location": str(D / "service.key")}
SCRAM = {"bootstrap.servers": (D / "bootstrap-sasl").read_text(),  # user name and password
         "security.protocol": "SASL_SSL", "ssl.ca.location": str(D / "ca.pem"),
         "sasl.mechanisms": "SCRAM-SHA-256", "sasl.username": "booknest-analytics",
         "sasl.password": (D / "analytics.password").read_text()}

aiven_producer.py is Producing Order Events's producer with **CERT in place of localhost:33092, and aiven_analytics.py is Consuming for Analytics's analytics consumer with **SCRAM, which finally tries to write one record as the same user. Run from WSL 6 , against Frankfurt:

Output of 92

$ python aiven_producer.py
390,737 sent, 0 failed, 0 undelivered in 111 s
$ python aiven_analytics.py
390,737 events in 63 s: {'order_placed': 100000, 'order_paid': 94106, 'order_cancelled': 5890,
  'order_shipped': 93803, 'order_delivered': 93009, 'order_returned': 3929}
orders 100,000, cancelled 5,890, revenue kept 3,270,268.02
write: Broker: Topic authorization failed
$ python aiven_admin.py
brokers: [1, 2] controller: 1
partition 0: leader 2, replicas [2, 1], isr [1, 2]
partition 1: leader 1, replicas [1, 2], isr [1, 2]
retention.ms=259200000
min.insync.replicas=1
...

All 390,737 sample events went up and came back, and the analytics consumer computed the same 100,000 orders and the same revenue, 3,270,268.02, as on the local broker in Consuming for Analytics: the logic did not change, only the address and credentials. Throughput was 3,500 events a second in and 6,200 out, well inside the plan's 250 KiB/s each way once zstd 126 shrank the stream (Compression Codecs). The read-only user could not write. The admin view shows settings you do not control on this plan: retention fixed at 72 hours (retention.ms=259200000, though the API reported 168 hours) and min.insync.replicas=1, so an acks=all write is acknowledged by one broker when the other is down. When the runs were captured, delete.py removed the service:

Output of 92

DELETE booknest-kafka: HTTP 200 deleted
GET booknest-kafka: HTTP 404 Service does not exist
booknest- services left: []

MENU