Pikhta Tech LLC · https://firai.ru
One method for four scenarios: benches, conditions, metrics, report forms. Applies to any vendor and any industry — manufacturing, logistics, retail, transport, energy, agriculture.
The test method is not tied to our devices: the same benches and the same metrics for any box and any camera. Compare us with anyone by the same rules.
Every scenario ends with a number: share of legible small text, share of frames without motion blur, connector failures after −25 °C, list of open ports.
Whoever writes the test method usually writes the tender requirements too. We suggest making the acceptance report part of your RFP (technical specification).
Measurements include conditions, limitations and stop reasons. Engineering results are available on the board test page.
| No. | Scenario | Bench and conditions | Metrics |
|---|---|---|---|
| 01 | Stationary indoor camera | 3 m test setup: shelving or a wall with small markings; illuminance 700 / 300 / 100 / 50 lx; four camera mounting positions (ceiling, head-on, on a pole, overhead); night mode without IR illumination | share of legible small text (markings, numbers, price tags); detector mAP on frames from each camera; effect of IR on object color |
| 02 | Fast motion in the frame | 200 passes of an object through the working zone: conveyor, scanning zone, manipulator, hand; swapped object, missed object; rolling vs global shutter compared on the same scene | share of frames without motion blur; detection accuracy; latency from frame to model result |
| 03 | Outdoor, machinery and vehicles | sun and headlights in the frame (WDR); night with IR illumination (snow, rain); vibration table / driving around the yard; water wash-down; −25 °C in a freezer for 2 hours and +50 °C in a closed cabinet; voltage sag (brownout) | legibility of license plates and markings in backlight; false alarms at night; connector failures and fogging; behavior under voltage sag |
| 04 | Security check of every camera and device we do this ourselves before handing over samples — and deliver the report together with the device | nmap of all ports; monitoring of outbound connections (vendor cloud, P2P); password change and UPnP disabled; model checked against the FSB notification registry and for an EAC declaration | list of open ports; list of external addresses (must be empty); ability to disable the cloud through standard settings |
Engineering tests of ROCK 5B+, ROCK 5C and ROCK 4D with saved model sources and fixed inputs. RK3576 models were rebuilt, with quality measured from fresh outputs. Default and configured modes are shown separately; cooling, OS, CPU affinity and frequency policies are part of each result's conditions. The SoC effect was not isolated from the OS and settings. YOLO26n speed: 640×640 input, batch 1; Runtime, ready BGR frame → detections and parallel contexts are measured separately. On 4D, runtime and ready-frame processing were also measured for segmentation, depth and OSNet on a ready crop. Main series: at least 60 s and 100 warmup frames, three 60 s measurements, median and all repeats. Short live trials and a single-context 30-minute run are separate. Quality is evaluated on fixed data with sample size stated.
Results, conditions and method: /en/testing/results/. Temperature is the maximum of built-in sensors; watchdog threshold 65 °C; ambient temperature unknown. Power draw, maximum video stream count and camera-to-display latency are unmeasured. This is not a box test at +40 °C in a cabinet or a completed Jetson comparison. For device acceptance, agree conditions, duration, power meter and temperature sensors separately before the run.
| Model | Input | Task |
|---|---|---|
| OSNet x0.25 | 256×128 (H×W) | ReID embeddings; FP16, with BoT-SORT |
| YOLO26n | 640×640 | detection; FP16 / INT8 / hybrid |
| YOLOv8n-seg | 640×640 | segmentation; FP16 / INT8 |
| Depth Anything V2 Small | 518×392 | relative depth; FP16 |
| DA3-Small | 518×392 | relative depth; FP16, adapted graph |
| Metric | Unit |
|---|---|
| throughput with measurement boundary stated | frames/s / crops/s |
| quality on the stated sample | px / IDF1 / AP / correct pairs |
| maximum of built-in sensors | °C |
| power draw — not measured | W |
| maximum video streams — not measured | streams |
| RKNN-Toolkit2 | 2.3.2 |
|---|---|
| RKNNLite / runtime | 2.3.0 |
| RKNPU | 0.9.8 |
TOPS for Rockchip are dense INT8 per the datasheet; for Jetson — sparse per the NVIDIA datasheet, converted to dense; the figures are not directly comparable, so comparison is by FPS on the same models. Comparative Jetson measurements are not yet available.
| No. | Check | Expected result |
|---|---|---|
| 1 | nmap of all TCP/UDP ports on the camera and the device | the list of open ports matches the product passport; no extra services |
| 2 | Outbound connection monitoring for 24 hours (traffic mirroring) | no external addresses: only your update server, your NTP and the addresses you set |
| 3 | Vendor cloud, P2P, UPnP | can be disabled by standard means and are disabled; do not come back after a reboot |
| 4 | Default passwords | changed; unique password per unit |
| 5 | FSB notification registry / EAC declaration | the model is listed in the registry or no notification is required (no radio); EAC declaration is available or in progress for the supplier |
| 6 | Device image: access only with the customer's keys, no vendor keys; outbound connections only to the specified addresses | verifiable on the sample; vendor security questionnaire — on request |
Scenario 04 metrics: list of open ports; list of external addresses (must be empty); ability to disable the cloud through standard settings. we do this ourselves before handing over samples — and deliver the report together with the device.
| Disabled | Bluetooth / Wi-Fi (physically absent on devices without radio), Avahi / mDNS, UPnP, telnet, default passwords, UART debug ports disabled on production units |
|---|---|
| Where it connects | only to the addresses in your config: your update server, your NTP, the services to which your container sends its output; you see and edit the list yourself |
| Where it doesn't | the internet, us, the SoC vendor; no telemetry to us or to the SoC vendor; we have no cloud and no message broker of our own |
| Updates | signed images, delivered from your network, automatic rollback on failure |
| Audit | list of open ports and component versions — in the vendor security questionnaire on request; for IP cameras — a check report for every unit at incoming inspection |
One form per scenario. Rows — the scenario's metrics; add your own if needed. Results are entered as numbers.
| Date and place | Site / scene | ||
|---|---|---|---|
| Product (vendor, model, serial no.) | Firmware / image version | ||
| Bench conditions | 3 m test setup: shelving or a wall with small markings; illuminance 700 / 300 / 100 / 50 lx; four camera mounting positions (ceiling, head-on, on a pole, overhead); night mode without IR illumination | ||
| No. | Metric / check | Result | Threshold (from the RFP) | Pass / fail |
|---|---|---|---|---|
| 1 | share of legible small text (markings, numbers, price tags) | |||
| 2 | detector mAP on frames from each camera | |||
| 3 | effect of IR on object color |
The pass / fail threshold is set in the customer's RFP or agreed before testing and entered in the "Threshold" column; the result is a number, not "satisfactory".
| Date and place | Site / scene | ||
|---|---|---|---|
| Product (vendor, model, serial no.) | Firmware / image version | ||
| Bench conditions | 200 passes of an object through the working zone: conveyor, scanning zone, manipulator, hand; swapped object, missed object; rolling vs global shutter compared on the same scene | ||
| No. | Metric / check | Result | Threshold (from the RFP) | Pass / fail |
|---|---|---|---|---|
| 1 | share of frames without motion blur | |||
| 2 | detection accuracy | |||
| 3 | latency from frame to model result |
The pass / fail threshold is set in the customer's RFP or agreed before testing and entered in the "Threshold" column; the result is a number, not "satisfactory".
| Date and place | Site / scene | ||
|---|---|---|---|
| Product (vendor, model, serial no.) | Firmware / image version | ||
| Bench conditions | sun and headlights in the frame (WDR); night with IR illumination (snow, rain); vibration table / driving around the yard; water wash-down; −25 °C in a freezer for 2 hours and +50 °C in a closed cabinet; voltage sag (brownout) | ||
| No. | Metric / check | Result | Threshold (from the RFP) | Pass / fail |
|---|---|---|---|---|
| 1 | legibility of license plates and markings in backlight | |||
| 2 | false alarms at night | |||
| 3 | connector failures and fogging | |||
| 4 | behavior under voltage sag |
The pass / fail threshold is set in the customer's RFP or agreed before testing and entered in the "Threshold" column; the result is a number, not "satisfactory".
| Date and place | Site / scene | ||
|---|---|---|---|
| Product (vendor, model, serial no.) | Firmware / image version | ||
| Bench conditions | nmap of all ports; monitoring of outbound connections (vendor cloud, P2P); password change and UPnP disabled; model checked against the FSB notification registry and for an EAC declaration | ||
| No. | Metric / check | Result | Threshold (from the RFP) | Pass / fail |
|---|---|---|---|---|
| 1 | list of open ports | |||
| 2 | list of external addresses (must be empty) | |||
| 3 | ability to disable the cloud through standard settings |
The pass / fail threshold is set in the customer's RFP or agreed before testing and entered in the "Threshold" column; the result is a number, not "satisfactory".
Pikhta Tech LLC · https://firai.ru · info@firai.ru · +7 (916) 877-93-11. The document is updated together with the site; current version — https://firai.ru/en/testing/. Test program v0.9 of 6 Oct 2026.