Zeabur's Breach Report and the Credentials Nobody Rotated
Zeabur and 91APP both published investigations in the past two weeks. In both cases, the attacker used keys that should have been replaced already, or that never belonged where they were stored.
Zeabur announced on October 17 that it would shut down at the end of the year, and published a preliminary investigation on the 19th. The report is detailed. It covers the attack path and the number of affected services. But the part I keep rereading is this: after the August incident, they rotated the keys that were confirmed leaked at the time, and they disabled the Legacy API Key. The cluster admin token and AWS keys stored in plaintext in the config repository were not in that rotation. Those are what the attacker used this time.
Two paths, joined at the end
According to Zeabur, the attacker came in on October 14 (UTC) through two separate paths.
The first started with an internal AWS key, then used a shared-cluster admin token. The token had no expiration, and the admin interface was exposed directly on the public internet. That made it possible to connect to the shared clusters in 9 regions from outside and read service configs and environment variables. Later, using another over-privileged key, the attacker read backups of an old cluster, worked their way into the secrets management service, and took about 53 production secrets. These included internal network admin credentials, platform database accounts, and the GitHub App private key.
The second path was the GCP dedicated servers. The initialization config for these machines stored the root password in plaintext. The attacker had an old GCP key that could read those configs, so they logged in as root on 101 machines and put their own SSH keys on 50 of them.
The two paths joined at the end. The attacker used the internal network credentials from the first path to enter the internal network from the machines in the second path, then connected to the platform database with a read-only account. Within 37 minutes they read more than 3.21 million environment variable records and 320,000 account records.
The report says the plaintext root password was discovered on September 3, but it was not fixed in time.
36 hours, no alerts
About 36 hours passed between the attacker's first use of the internal credentials and the team starting its investigation. No alert fired during that time. In the end, a user report is what revealed the problem.
I'm not saying this to make fun of anyone. Monitoring at small teams is often the last thing that gets added, and many people's systems probably wouldn't pass this kind of test either. The root causes in the report are fairly plain: credentials that last a long time and are stored in plaintext, an admin interface open to the outside, permissions that are too broad, and abnormal access that nobody noticed. Each one looks like a small thing on its own.
91APP followed a similar path
In the same week, 91APP published an 18-day investigation report on October 22. The entry point was Appsmith, a third-party internal tool with a known vulnerability that had not been patched. The attacker got into the internal network through it, then obtained a database access key from GitLab and read an order database for Commerce Cloud. The investigation team found a record of an unauthorized execution thread on an internal CI host dated October 3, which let them reconstruct the whole path. Traces of ARTEX, an AI-driven autonomous penetration tool, were also found in the system.
Afterward, they removed Appsmith, replaced internal tokens and database passwords, rebuilt GitLab permissions, and restricted connections between the CI host and the database network segment.
The details of the two incidents differ, but in both cases the damage grew after the first breach. Once inside, the attacker relied on keys that were stored in the wrong place, lived too long, or had too much access.
Checking this site against it
This site is small, so the comparison is quick. The frontend build uses a single read-only role, and that role can only SELECT from a view that contains published posts. So even if the build environment's connection string leaked, the only thing readable would be content that is already public. The backend Worker verifies the Cloudflare Access JWT on every request itself, and does not assume that "Access is in front, so it's fine." The Deploy Hook and AI service keys live in the deployment environment, not in Git.
These were part of the plan from the start. But Zeabur's report made me think about another question: on the day something really went wrong, where would I start rotating? If the list is built by working outward from "the things we know leaked," it is easy to miss anything sitting in config files, backups, or initialization scripts. A safer approach is to first inventory every place where credentials are stored.
If you happen to have services on Zeabur, the official advice is to replace all credentials in environment variables, log in again, and rebuild platform access tokens. Shared-cluster users need to change their passwords inside the database. Dedicated GCP server users need to back up and rebuild the machines, because a backdoor may remain on them. Some services will begin shutting down on December 1, and all dedicated servers and shared clusters will be shut down before December 31.
Sources:
- Zeabur Publishes Breach Investigation, Attacker Read More Than 3.21 Million Environment Variables (iThome, in Chinese)
- Investigation of the 91APP Breach Finds ARTEX Attack Traces (iThome, in Chinese)