# ZKTeco Deployment Incident Lessons

Date: 2026-07-30

## What happened

We changed the Laravel-to-Flask authentication and device-health flow, then
deployed parts of it manually. Several separate problems looked like one
failure:

- Flask was running on a Windows EC2 instance as an NSSM service, not as a
  foreground Git Bash process.
- More than one Flask-related Windows service/process existed, and two
  processes briefly listened on port 80.
- The EC2 public IP had changed. The old address was
  `18.203.250.190`; the active address was `3.254.49.131`.
- `setx ... /M` did not work correctly when called through Git Bash. The
  machine environment variable was set with PowerShell instead.
- Restarting `run_server.bat` in a terminal created a confusing foreground
  process while NSSM was also managing Flask.
- Python tests were run with system Python, which did not contain
  `pyzkaccess`; the project virtual environment did.
- A valid shared token proved that Laravel could reach Flask, but it did not
  prove that Flask could reach a controller.
- ICMP ping succeeded for one controller while TCP port `4370` failed.

## The important diagnostic distinction

Always test the path in layers:

```text
Laravel -> Flask HTTP/authentication -> controller TCP port -> ZKTeco SDK
```

Interpret responses this way:

- `401 Unauthorized`: shared secret or cached Laravel/Flask configuration is
  wrong.
- `200` with `success: false`: authentication worked; the controller path or
  SDK connection failed.
- HTTP timeout: Flask or the controller path did not answer in time.
- ICMP success with TCP failure: the host responds to ping, but the ZKTeco
  service is not reachable on its configured port.

The successful test for the incident was:

```text
HTTP 200 + {"success": false}
```

That ruled out the shared-secret problem for that request. The controller then
failed the independent AWS-side test:

```text
PingSucceeded: True
TcpTestSucceeded: False
```

## Windows Flask deployment lessons

Find the service owner before starting anything manually:

```bash
powershell.exe -NoProfile -Command 'Get-CimInstance Win32_Service | Where-Object { $_.PathName -match "nssm" } | Select Name,State,ProcessId,PathName'
```

Inspect the active service:

```bash
/c/nssm-2.24/win64/nssm.exe get FlaskAPI Application
/c/nssm-2.24/win64/nssm.exe get FlaskAPI AppDirectory
/c/nssm-2.24/win64/nssm.exe get FlaskAPI AppParameters
```

For this server, `FlaskAPI` owns `run_server.bat`. Deploy the files, then
restart the service:

```bash
powershell.exe -NoProfile -Command 'Restart-Service -Name "FlaskAPI" -Force'
```

Do not start `cmd.exe //c run_server.bat` in a second terminal while NSSM is
running it. That can create duplicate listeners and makes it unclear which
code or environment is serving requests.

Set a machine-level secret from Git Bash without printing it:

```bash
read -rsp "Enter secret: " S; echo
export ZKTECO_SHARED_SECRET="$S"
powershell.exe -NoProfile -Command '[Environment]::SetEnvironmentVariable("ZKTECO_SHARED_SECRET",$env:ZKTECO_SHARED_SECRET,"Machine")'
unset S
powershell.exe -NoProfile -Command 'Restart-Service -Name "FlaskAPI" -Force'
```

Never put the real secret in a committed file, a screenshot, or a command
that will remain in shell history.

## Laravel deployment lessons

Laravel configuration is cached, and Horizon workers are long-lived. After
changing `.env`:

```bash
php artisan optimize:clear
php artisan config:cache
php artisan horizon:terminate
```

The same exact non-empty `ZKTECO_SHARED_SECRET` must be configured on Laravel
and Flask. Check only whether it exists and its length; do not print its value:

```bash
php artisan tinker --execute='dump([
  "bridge" => config("ltdgroup.zkteco_ip"),
  "secret_configured" => filled(config("ltdgroup.zkteco_shared_secret")),
  "secret_length" => strlen((string) config("ltdgroup.zkteco_shared_secret")),
]);'
```

`php artisan zkteco:check-status` reports that health jobs were queued. It does
not mean that they completed. Horizon must have an active supervisor for the
`device-health` queue.

## Testing lessons

Run Python integration tests from the Flask repository with its virtual
environment:

```bash
cd ~/code/ltdgroup/elevator-mon/ZKT-access-flask
PYTHONPATH=. .venv/bin/python -m unittest discover -s tests -p 'test_*integration.py'
```

System Python can report `ModuleNotFoundError: pyzkaccess` even though the
project environment is correct. Keep `.venv/` ignored by Git.

## Safe retry rule

Do not retry all failed Horizon jobs immediately. First verify:

1. Laravel and Flask accept the same shared secret.
2. The bridge public IP is current.
3. The relevant controller IP and port are correct.
4. AWS can reach that controller over TCP.
5. Horizon is consuming the intended queue.

Then retry one known failed job first:

```bash
php artisan queue:retry JOB-UUID
```

## Prevention checklist

- Record the service manager and exact process command for every production
  bridge.
- Verify the public IP before changing Laravel `ZKTECO_IP`.
- Test `/` for liveness, then test an authenticated controller endpoint.
- Test the actual controller TCP port from the bridge host.
- Deploy Laravel config and restart Horizon together.
- Deploy Flask code and restart NSSM together.
- Confirm one listener on port 80.
- Check one health result before retrying tag operations.

## One-line takeaway

An authenticated bridge is not the same as a reachable controller, and a
queued job is not the same as a completed health check.