MSK Provisioned Clusters

Amazon Managed Streaming for Apache Kafka 129 (MSK) provisioned clusters are Apache Kafka brokers that AWS 24 creates in your VPC, spreads over two or three availability zones, patches, monitors and replaces when they fail. You choose the Kafka version, broker size and count, and still own topics, partitions and most broker settings, through a cluster configuration applied with rolling restarts. Standard brokers (kafka.m7g.large and up) are classic Kafka on EBS volumes you provision, with optional tiered storage (Tiered Storage); Express brokers (express.m7g.large and up) are AWS's reworked brokers with storage billed by use, for which AWS claims up to three times the throughput per broker and 20 times faster scaling.

Versions trail Apache closely: MSK offers up to 4.2.x (since 15 July 2026, on Express brokers only, where Queues for Kafka of Consumers and Queues is not yet supported), with 3.9.x marked "recommended". AWS's US East examples price a kafka.m7g.large at $0.204 per broker-hour and storage at $0.10 per GB-month; an express.m7g.large at $0.408 per hour plus $0.01 per GB written; tiered storage at $0.06 per GB-month plus $0.0015 per GB read back. Three Standard brokers with 100 GB come to about $465 a month for BookNest, almost all of it for brokers that would sit idle: provisioned MSK pays off at steady, substantial throughput.

Clients authenticate with mutual TLS, SASL/SCRAM (secrets in AWS Secrets Manager 24 ) or IAM, which needs no passwords: Java clients may use AWS's AWS_MSK_IAM mechanism, and every client can use standard SASL/OAUTHBEARER with a token signed from its AWS credentials by AWS's aws-msk-iam-sasl-signer library. In librdkafka-based clients such as BookNest's Python producer, that is a token callback. The same client configuration runs here against a local Kafka 4.3.1 broker (demos/ch06/cloud/oauth_up.sh) whose listener accepts OAUTHBEARER with Kafka's development-only unsecured JWT validator, so the callback builds an unsigned token where MSK's would sign one:

oauth_client.py (excerpt): an OAUTHBEARER producer, as MSK IAM clients configure itPython

def token(_oauth_config):         # on MSK: MSKAuthTokenProvider.generate_auth_token(region)
    now = int(time.time())
    claims = {"sub": "booknest-producer", "iat": now, "exp": now + 900}
    return b64({"alg": "none"}) + "." + b64(claims) + ".", claims["exp"]
conf = {"bootstrap.servers": "localhost:32192",     # on MSK: the IAM bootstrap string
        "security.protocol": "SASL_PLAINTEXT",     # on MSK: SASL_SSL
        "sasl.mechanisms": "OAUTHBEARER",
        "oauth_cb": token,                         # called again before each token expires
        "error_cb": lambda err: print("error:", err)}

Output

$ python oauth_client.py
delivered to booknest.order-events [2] at offset 0
unflushed: 0
$ python oauth_client.py plain
error: KafkaError{code=_AUTHENTICATION,val=-169,str="sasl_plaintext://localhost:32192/bootstrap
  :
SASL PLAIN mechanism handshake failed: Broker: Unsupported SASL mechanism: broker's supported
mechanisms: OAUTHBEARER (after 305ms in state AUTH_HANDSHAKE)"}
...
unflushed: 1

The client calls the callback again before each token expires, so short-lived IAM tokens never interrupt a producer. With PLAIN, the handshake fails and the broker names the mechanisms it accepts: the first thing to read when a cloud client cannot connect.

MENU