# Operator alerts
Every line monetize writes for its operator starts `ALERT <code>`. The ONE box hears it
(`monetize_admin::alerts`), keeps it by its code and mails it to the operator through the
box's own relay (`secret.alert_to`): the first mail at once, then at most one per ten
minutes per code, so nothing is lost and a storm stays one mail. `code-red` never waits.
`ALERT:` with a colon inside an order's history is a note on that order, not a log line.
`crates/monetize-admin/tests/the_alert_table.rs` holds this table to the code: every
`ALERT <code>` monetize can write is a row, every row is a code it writes, and an
`ALERT` with no code fails the build.
Severity:
- **critical**: money or a customer's data is at risk, or a server refuses to serve. Act now.
- **warning**: something is stuck that will not fix itself. Act today.
- **notice**: said and retried by itself. Look if it repeats.
Who: **operator** is the ONE box's operator (the mail). **operator, at once** is a
CODE RED: it is mailed with no pacing and stays raised until the step that clears it
succeeds.
## Codes monetize writes
| code | severity | who | what it means | what to do |
|---|---|---|---|---|
| `code-red` | critical | operator, at once | A step that must not fail failed (a custody take-back, an unsaved run): the line names the site, what broke, the error and the next step. | Do what its `next:` says; the next success clears it. |
| `attach.bearer_expiring` | warning | operator | An estate's ONE bearer stops opening Signup, Purchase and EstateOrder in a few days. | Enrol the estate again (an attach token, then the kit's attach) before the date. |
| `attach.bearer_expired` | critical | operator | An estate's ONE bearer has expired: its signups and purchases are refused. | Enrol the estate again now. |
| `joined.transient` | warning | operator | A joined estate's record could not be read or taken this tick (the guardian store refused a write, the record was unreadable): it stays and every tick tries again (A32-1). | Look at the ONE box's disk and its guardian store; restart monetize if the store stays refused. |
| `console.fleet-job-failed` | warning | operator | The console's fleet job (the ONE `Guardian.Cases` read its pages show, RA3-1) failed: the pages show the last fleet it read, with its age. | Look at the ONE box's monetize: is it up, does the console's bearer still open `Guardian`? The job asks again every 30 s. |
| `cloud-guest.state-unwritten` | warning | operator | A cloud guest's state was not written after an op; the answer went back as it was. | It is retried on the next call; if it repeats, check the guest state store's disk. |
| `entitlement.push_stuck` | warning | operator | An entitlement has not reached its product after many pushes. | Check the product's door and its bearer; it is still pushed every tick. |
| `entitlement.site_not_holding` | warning | operator | A paid fact (a paid plan, its term) is on the ledger, but the site's main does not hold it, so the buyer's plan reads wrong there (ALICA-4). The ONE box queues it to the site again on every tick. | Nothing, if it stops after the site's next pull; if it repeats, check that the site's main pulls from this box (its pull slot and bearer). |
| `estate-orders.account-not-made` | critical | operator | An order was paid and its account could not be made. | Make the account right by hand; nothing is laid for another's account. |
| `estate-orders.account-row` | notice | operator | An order's row on its account was not written. | The next tick tries again. |
| `estate-orders.custody` | critical | operator | An estate's secrets were not forgotten after the hand-off. | Remove them from the custody by hand. |
| `estate-orders.dns-by-hand` | warning | operator | A DNS record must be set by hand (no writer for the zone). | Set the record the line names. |
| `estate-orders.dns-not-found` | critical | operator | A give-back did not find the DNS records its lay wrote where they were written, or the writer that wrote them is no longer held. | Remove the records the line names by hand, in whichever zone they stand. |
| `estate-orders.dns-takeover` | warning | operator | A lay took over existing DNS names of the operator's site. | Check that the names should point at the new lay. |
| `estate-orders.dns-writer` | critical | operator | The shop has no usable DNS writer: nothing is sold where it writes. | Feed or fix the zone's writer. |
| `estate-orders.guardian` | notice | operator | The guardian could not be asked about an order's estate. | It is asked again on the next tick. |
| `estate-orders.install-abandoned` | critical | operator | An install was given up. | Settle the order (retry or refund). |
| `estate-orders.install-deadline` | warning | operator | An install passed its deadline. | Look at the install's log; settle it if it does not finish. |
| `estate-orders.queued-deadline` | critical | operator | A paid order waited in the queue for three days with no room to set it up: it is owed now (A32-22). | Set it up again or refund it (`site settle`). |
| `estate-orders.queued-long` | warning | operator | A paid order has waited in the queue for six hours (told again every six while it waits); its buyer reads the day it is set up or refunded by (A32-22). | Raise the cap or the slots, or settle it. |
| `estate-orders.material` | warning | operator | An estate never took its installer material and it was forgotten. | The estate cannot be upgraded, recharged or renewed by itself; re-lay or settle it. |
| `estate-orders.owed` | critical | operator | A paid order failed to be set up and waits for the operator. | Retry it, or cancel it with a refund. |
| `estate-orders.stopped` | warning | operator | An order's terms are refused for good (its site left the feed, sells another product, or names none): the run is stopped by name, neither booked nor driven, looked at again every 10 min. Nothing of it is given back or billed. | Declare its site in the feed again (it resumes by itself), or terminate it. |
| `estate-orders.page` | critical | operator | A payment page was not closed at the vendor, or a payment was captured for nothing. | Close the page, or refund the capture, at the vendor. |
| `estate-orders.queue` | warning | operator | An order waits in the cloud's queue longer than it should. | Check the cloud's capacity and the order's words. |
| `estate-orders.ready` | warning | operator | The first-login invite could not be read, so the ready mail waits. | Check the estate's invite; the mail goes when it can be read. |
| `estate-orders.refund-forget` | warning | operator | A refunded order's custody could not forget its sealed pairing material. | Remove the material from the custody by hand. |
| `estate-orders.reversal` | critical | operator | A vendor reversed a payment (a refund or a dispute) on an order. | Check the order and its estate; take back what the line says. |
| `estate-orders.terminate` | critical | operator | A termination step failed or could not take the run. | Look at the order; the next tick asks again. |
| `estate-orders.tick` | critical | operator | An estate-order tick panicked. | Read the panic; report it as a bug. |
| `estate-orders.unwritten` | warning | operator | An order's run changed and could not be written. | The step runs again; check the order store's disk if it repeats. |
| `estate-orders.update` | warning | operator | An estate's update step could not go on. | Read the words; run the update again when they are fixed. |
| `front.bearer-missing` | critical | operator | The front has no bearer and refuses to serve. | Write the bearer file the line names. |
| `grow.failed` | warning | operator | A grow failed. | Read why; the room stays as it was. |
| `grow.not_told` | critical | operator | A sale went past the room and the estate could not be told to grow. | Grow the estate by hand or fix its door. |
| `grow.twin-alone` | critical | operator | The main grew and its twin did not catch up. | Grow or recover the twin. |
| `guardian.looks-refused` | warning | operator | jera refused the guardian's provider-looks job as stale (RA3-2): no look at the provider was asked this tick. | It should not happen (one key, one generation); if it repeats, restart monetize and tell the lead lane. |
| `guardian.case-unsaved` | warning | operator | A guardian case could not be saved. | The next tick moves it again from the store. |
| `guardian.operator_rebuild` | critical | operator | A guardian case waits for the operator to rebuild a box in the estate's own cloud. | Rebuild the box with the product's own ops kit. |
| `money-log.unwritten` | warning | operator | PID 1 could not write the money lines' own home (`<data>/money/lines.log`, ALICA-3); said once. The lines are still in the boot log, where noise can push them out. | Check the data disk (full, read-only); the next money line is tried again. |
| `jobs.failed` | warning | operator | A background job of the server (a jera job: a wallet or plan reading, a grow drive, a callback's settle or grant, a keeper, the guest watch, the off-box copy) failed or panicked; the line names its kind and subject. | Fix what it names; a duty's next look runs as due, a one-shot's subject is asked again by its caller. |
| `jobs.unstarted` | warning | operator | A keeper duty of the server could not start (no runtime). | Restart the server; until then the readings are made when a customer asks. |
| `jobs.unrecorded` | warning | operator | The server's (or PID 1's) jobs' ledger under `<data>/jobs` did not open: every job still runs, but none is a row in `nornir jobs` (RA4-18). | Read the error (a full or read-only disk, a second process holding the ledger); restart once it is fixed. |
| `guests.watch-unstarted` | warning | operator | The guest watcher did not start. | Restart the server after a new or rebuilt guest. |
| `joined.unstamped` | critical | operator | A refused joined record could not be marked as refused in itself; it was moved aside as it is (RA1-6). | Check the data disk; write the reason into `<estate>.json.refused.why` by hand before the next restart, or a later app may read it as an older app's and put it back. |
| `joined.why-lost` | warning | operator | The reason a joined record was refused could not be written beside it; the record is kept aside with its reason in itself (RA1-6). | Check the data disk (full, read-only); nothing is put back. |
| `narrate.no-receiver` | notice | operator | This build links no dev-narrate receiver. | Nothing, unless you meant to build `narrate`. |
| `nonpay.deleting` | notice | operator | A product is purging a non-paying account. | Nothing; it is the nonpay engine's step. |
| `nonpay.delete_failed` | critical | operator | A non-paying account's deletion did not complete. | It is retried in an hour; check the product if it repeats. |
| `offsite.failed` | warning | operator | The off-site backup failed. | Fix what it names; the next run tries again. |
| `orders-waiting` | warning | operator | Grow orders wait for room in the cloud. | Check the cloud's capacity. |
| `plugin.refused` | warning | operator | A plugin update or look was refused. | Read the plugin row; fix the plugin or its pin. |
| `product-guest.facts-unread` | critical | operator | A product guest's facts could not be read: its tickets, test buttons and guardian are off. | Fix the guest; it is asked again on the next look. |
| `self-update.confirm` | warning | operator | A new release serves, but its slot could not be confirmed. | Confirm the slot, or it is rolled back. |
| `self-update.reexec` | critical | operator | A self-update's re-exec failed and its children are stopped. | Start the box again; read the error. |
| `self-update.refused` | warning | operator | A self-update was refused. | Read why; the box runs the release it had. |
| `self-update.rollback` | critical | operator | A new release did not serve in time and was rolled back. | Read the release's log before trying again. |
| `self-update.rolled-back` | warning | operator | The box runs a rolled-back release. | Read why before the next update. |
| `server.bearer-missing` | critical | operator | monetize-server has no bearer and refuses to serve. | Write a bearer to the token file (0600). |
| `server.bearer-refused` | critical | operator | monetize-server was asked to serve open while a bearer is configured. | Drop the open flag. |
| `sold-estate.grow` | warning | operator | A grow's go-ahead was not queued for the estate's boxes. | Read why; ask again. |
| `sold-estate.keep-boot-unsaved` | warning | operator | A box was keep-booted to a release and the run was not saved. | Check the run; the step is recorded when it is done. |
| `spend-cap` | warning | operator | A grow was refused by the spend ceiling. | Raise the ceiling, or leave the product at its size. |
| `twin-grow.journal` | warning | operator | A twin grow's order could not be journaled. | A restart may order it again; check the journal's disk. |
| `users.refunded` | critical | operator | A purchase was paid and could not be done, so it was refunded. | Tell the buyer; check why. |
## Kinds the box makes of other lines
| code | severity | who | what it means | what to do |
|---|---|---|---|---|
| `critical` | critical | operator | A line that says CRITICAL. | Read the line. |
| `grow.room_queued` | notice | operator | A sale past the room was queued for its estate. | Nothing; the grow follows. |
| `payment.refused` | warning | operator | A callback, purchase, checkout, capture, refund or vendor call was refused. | Read the refusal. |
| `attach.refused` | warning | operator | An estate's enrolment or install notice was refused. | Read the refusal; enrol again. |
| `case.<state>` | notice | operator | A guardian case moved (a recharge, a rebuild, a fence). | Follow the case. |