README
¶
nethsecurity-controller
Build
CGO_ENABLED=0 go build
Environment variables
Mandatory
-
ADMIN_USERNAME: admin username to login -
ADMIN_PASSWORD: admin password to login -
SECRET_JWT: secret to sing JWT tokens -
REGISTRATION_TOKEN: secret token used to register units -
CREDENTIALS_DIR: directory to save credentials of connected units -
PROMTAIL_ADDRESS: promtail address -
PROMTAIL_PORT: promtail port -
PROMETHEUS_PATH: prometheus web path -
WEBSSH_PATH: webssh web path -
GRAFANA_PATH: grafana web path -
GRAFANA_POSTGRES_PASSWORD: password to access grafana postgres database -
REPORT_DB_URI: Timescale database URI for reports
Optional
-
LISTEN_ADDRESS: a comma-separated list of listen addresses for the server. Each entry is in the form<address>:<port>- default:127.0.0.1:5000Example:127.0.0.1:5000,192.168.100.1:5000 -
OVPN_DIR: openvpn configuration directory - default:/etc/openvpn -
OVPN_NETWORK: openvpn network address - default:172.21.0.0 -
OVPN_NETMASK: openvpn netmask - default:255.255.0.0 -
OVPN_UDP_PORT: openvpn udp port - default:1194 -
OVPN_C_DIR: openvpn path of ccd directory - default: OVPN_DIR +/ccd -
OVPN_P_DIR: openvpn path of proxy directory - default: OVPN_DIR +/proxy -
OVPN_K_DIR: openvpn path of pki directory - default: OVPN_DIR +/pki -
OVPN_M_SOCK: opevpn management socket path - default: OVPN_DIR +/run/mgmt.sock -
EASYRSA_PATH: easyrsa command path - default:/usr/share/easy-rsa/easyrsa -
PROXY_PROTOCOL: traefik protocol - default:http:// -
PROXY_HOST: traefik host - default:localhost -
PROXY_PORT: traefik port - default:8080 -
LOGIN_ENDPOINT: unit login endpoint, on stand-alone api server - default:/api/login -
FQDN: fully qualified domain name of the machine - default:hostname -f -
CACHE_TTL: cache time to live for unit information in seconds - default:7200(2 hours) Unit information are fetched from the connected units. The cache is refreshed every hour. -
RETENTION_DAYS: configure how many days the metrics should be kept - default:60 -
MAXMIND_LICENSE: license key for maxmind geolite2 database - default: `` If the license key is not set, the geolite2 database will not be downloaded. -
GEOIP_DB_DIR: directory to save geolite2 database - default: current directory -
SENSITIVE_LIST: list of sensitive information to be redacted in logs -
VALID_SUBSCRIPTION: valid subscription status - default:false -
ENCRYPTION_KEY: key to encrypt/decrypt sensitive data, it must be 32 bytes long -
PLATFORM_INFO: a JSON string with platform information, used to store the controller version and other information. It can be left empty. Example:{"vpn_port":"1194","vpn_network":"192.168.100.0/24", "controller_version":"1.0.0", "metrics_retention_days":30, "logs_retention_days":90} -
PROMETHEUS_AUTH_PASSWORDandPROMETHEUS_AUTH_USERNAME: credentials to access the/prometheus/targetsendpoint for listing connected units in Prometheus target format. - default:prometheus:prometheus
User and units authorizations
A unit is a NethSecurity firewall that is connected to the controller.
Units are identified by a unique ID and are stored in the database. VPN info of the unit are stored in a file in the OVPN_DIR directory.
A group of units is a collection of units that can be managed together. Units can be added to a group via the API. The group is identified by a unique ID and is stored in the database.
A user is an account that can access the API and the UI. User accounts are stored in the database and can be managed via the API. By default, a user can't access any unit. A user can be promoted to an admin user, which allows the user to manage other users and units.
Admin users have the admin flag set to true and can:
- create, modify and delete user accounts
- create, modify and delete units
- create, modify and delete units groups
- assign a user to one or more groups of units
- see all units, despite the groups assigned to the user
The following rules apply:
- a user can be assigned to one or more groups of units, if the user is not assigned to any group, the user can't see any unit
- a non existing-unit cannot be added to a group
- a unit group that is associated to a user account can't be deleted
- a non-existing unit group cannot be assigned to a user account
- when a unit is deleted from the database, it is removed from all groups
Database design
The database is designed to efficiently manage and report on a large number of NethSecurity firewall units connected to a central controller. Each unit is uniquely identified and related data is stored in several reporting tables (such as dpi_stats, ovpnrw_connections, ts_attacks, etc.), supporting analytics and monitoring via TimescaleDB continuous aggregates.
Key design choices include:
-
No Foreign Keys:
Foreign key constraints (especially withON DELETE CASCADE) were removed from all tables. With high-volume tables likedpi_stats(which can grow by millions of rows per day on a controller with 70 units), cascading deletes caused severe performance issues—deleting a unit could take hours due to the volume of related data. -
Orphaned Data Cleanup:
Instead of relying on foreign keys, a stored procedure (cleanup_orphaned_unit_data) is scheduled to run daily. This procedure deletes all records from reporting tables where theuuidis not present in theunitstable, ensuring data consistency and preventing orphaned records. -
Indexes on UUID:
All reporting tables have an index on theuuidfield to speed up deletion and lookup operations. For thedpi_statstable, the index is only created on new (empty) installations, as creating it on an existing, large table would be prohibitively slow. -
TimescaleDB Features:
The schema leverages TimescaleDB hypertables and continuous aggregates for efficient time-series data storage and reporting, with retention policies to automatically drop old data. Queries can be perfomed on aggregated tables even if the original data is deleted, as the continuous aggregates are updated periodically.
APIs
Auth
-
GET /healthREQ
Content-Type: application/jsonRES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "status": "ok" } -
POST /loginREQ
Content-Type: application/json { "username": "root", "password": "Nethesis,1234" }RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "expire": "2023-05-25T14:04:03.734920987Z", "token": "eyJh...E-f0" } -
POST /logoutREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200 } -
GET /refreshREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "expire": "2023-05-25T14:04:03.734920987Z", "token": "eyJh...E-f0" }
Units
-
GET /unitsREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": [ { "ipaddress": "172.23.21.3", "id": "<unit_id>", "netmask": "255.255.255.0", "groups": ["group1", "group2"], "vpn": { "bytes_rcvd": "21830", "bytes_sent": "5641", "connected_since": "1686312722", "real_address": "192.168.122.220:41445", "virtual_address": "172.23.21.3" }, "info": { "unit_name": "myfw1", "version": "8-23.05.2-ns.0.0.2-beta2-37-g6e74afc", "subscription_type": "enterprise", "system_id": "XXXXXXXX-XXXX", "ssh_port": 22, "fqdn": "fw.local", "api_version": "1.0.0" } }, ... { "ipaddress": "", "id": "<unit_id>", "netmask": "", "vpn": {}, "groups": [], "info": { "unit_name": "", "version": "", "subscription_type": "", "system_id": "", "ssh_port": 0, "fqdn": "", "api_version": "1.0.0" } } ], "message": "units listed successfully" }The API takes a query parameter
cache. Ifcacheis set totrue, the API will return the cached data, if data are fresh enough. Ifcacheis set tofalse, the API will always fetch the data from the connected units. -
GET /units/<unit_id>REQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "ipaddress": "172.23.21.3", "id": "<unit_id>", "netmask": "255.255.255.0", "registered": true, "vpn": { "bytes_rcvd": "22030", "bytes_sent": "5841", "connected_since": "1686312722", "real_address": "192.168.122.220:41445", "virtual_address": "172.23.21.3" }, "info": { "unit_name": "myfw1", "version": "8-23.05.2-ns.0.0.2-beta2-37-g6e74afc", "subscription_type": "enterprise", "system_id": "XXXXXXXX-XXXX", "ssh_port": 22, "fqdn": "fw.local", "api_version": "1.0.0" }, "join_code": "eyJmcWRuIjoiY29udHJvbGxlci5ncy5uZXRoc2VydmVyLm5ldCIsInRva2VuIjoiMTIzNCIsInVuaXRfaWQiOiI5Njk0Y2Y4ZC03ZmE5LTRmN2EtYjFjNC1iY2Y0MGUzMjhjMDIifQ==" }, "message": "unit listed successfully" }The API takes a query parameter
cache. Ifcacheis set totrue, the API will return the cached data, if data are fresh enough. Ifcacheis set tofalse, the API will always fetch the data from the connected units. -
GET /units/<unit_id>/infoREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "fqdn": "NethSec", "ssh_port": 22, "subscription_type": "enterprise", "system_id": "XXXXXXXX-XXXX", "unit_name": "NethSec", "version": "NethSecurity 8 23.05.3-ns.1.0.1", "api_version": "1.0.0" }, "message": "unit info retrieved successfully" }The backend stores data inside the database. This is useful for retrieving new information of the unit without waiting for cron to store it.
-
GET /units/<unit_id>/tokenREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "expire": "2023-06-10T12:23:39.46160793Z", "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...rRikiEG83smBWPdHWzzhKOnfgzOkRXQntxdKGdaIhk8" }, "message": "unit token retrieved successfully" } -
POST /unitsREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN> { "unit_id": "<unit_id>" }RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "join_code": "eyJmcWRuIjoiY29udHJvbGxlci5ncy5uZXRoc2VydmVyLm5ldCIsInRva2VuIjoiMTIzNCIsInVuaXRfaWQiOiI2OThhMDQzZC02MGRiLTQyNmMtODRjZi1lODZhMTZmM2QxMzMifQ==" }, "message": "unit added successfully" } -
POST /units/registerREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN> { "unit_name": "fw.nethsecurity.local", "unit_id": "d330b2db-cdfe-4c56-b9b6-f97e5b838748", "username": "test", "password": "Nethesis,1234", "version": "8-23.05.2-ns.0.0.2-beta2-37-g6e74afc", "subscription_type": "enterprise", "system_id": "XXXXXXXX-XXXX" }RES | unit previously added
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "ca": "-----BEGIN CERTIFICATE-----\n\n-----END CERTIFICATE-----", "cert": "Certificate:\n\n-----END CERTIFICATE-----", "host": "ns8.local", "key": "-----BEGIN PRIVATE KEY-----\n\n-----END PRIVATE KEY-----", "port": "1194", "promtail_address": "172.21.0.1", "promtail_port": "5151", "api_port": "20001", "vpn_address": "192.168.0.1" }, "message": "unit registered successfully" }RES | unit not added in waiting list
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 403, "data": "", "message": "unit added to waiting list" } -
DELETE /units/<unit_id>REQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": "", "message": "unit deleted successfully" }
Unit Groups
-
GET /unit_groupsREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "unit_groups": [ { "id": 1, "name": "Group 1", "description": "This is a test group", "units": ["unit_id_1", "unit_id_2"], "created_at": "2024-03-14T09:37:28+01:00", "updated_at": "2024-03-14T10:00:00+01:00", "used_by": ["account_id_1", "account_id_2"] } ... ] }, "message": "unit groups listed successfully" } -
GET /unit_groups/<group_id>REQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "unit_group": { "id": 1, "name": "Group 1", "description": "This is a test group", "units": ["unit_id_1", "unit_id_2"], "created_at": "2024-03-14T09:37:28+01:00", "updated_at": "2024-03-14T10:00:00+01:00" } }, "message": "success" } -
POST /unit_groupsREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN> { "name": "Group 1", "descrption": "This is a test group", "units": ["unit_id_1", "unit_id_2"] }RES
HTTP/1.1 201 OK Content-Type: application/json; charset=utf-8 { "code": 201, "data": {"id": 1}, "message": "success" } -
PUT /unit_groups/<group_id>REQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN> { "name": "Group 1 updated", "description": "This is an updated test group", "units": ["unit_id_1", "unit_id_3"] }RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": null, "message": "success" } -
DELETE /unit_groups/<group_id>REQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": "", "message": "success" }
Accounts
-
GET /accountsREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "accounts": [ { "id": 2, "username": "test1", "password": "", "admin": true, "display_name": "Test 1", "created": "2024-03-14T09:37:28+01:00" }, ... { "id": 6, "username": "test2", "password": "", "admin": false, "display_name": "Test 2", "created": "2024-03-14T11:43:33+01:00" } ], "total": 5 }, "message": "success" } -
GET /accounts/<account_id>REQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "account": { "id": 2, "username": "test3", "password": "", "admin": false, "display_name": "Test 3", "created": "2024-03-14T09:37:28+01:00" } }, "message": "success" } -
POST /accountsREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN> { "username": "test1", "password": "Nethesis,1234", "display_name": "Test 1", "unit_groups": [1, 2], "admin": false }RES
HTTP/1.1 201 OK Content-Type: application/json; charset=utf-8 { "code": 201, "data": {"id": 5}, "message": "success" } -
PUT /accounts/<account_id>REQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN> { "password": "Nethesis,4321", "display_name": "Test 5", "unit_groups": [1, 2], "admin": false }RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": null, "message": "success" } -
PUT /accounts/passwordREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN> { "old_password": "Nethesis,1234", "new_password": "Nethesis,4321" }RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": null, "message": "success" } -
DELETE /accounts/<account_id>REQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": "", "message": "success" } -
GET /accounts/ssh-keysREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "key": "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNza...m3XHi7DiRCmyqbwdp86eV\n-----END OPENSSH PRIVATE KEY-----", "key_pub": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVled...UxVF6O0Esc3gFe0XMUT9Y+GtqM1O2s= [email protected]" }, "message": "success" } -
POST /accounts/ssh-keysREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN> { "passphrase": "Nethesis,2222" }RES
HTTP/1.1 201 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "key_pub": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVled...UxVF6O0Esc3gFe0XMUT9Y+GtqM1O2s= [email protected]" }, "message": "success" } -
DELETE /accounts/ssh-keysREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": "", "message": "success" } -
GET /platformREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "vpn_port": "1194", "vpn_network": "172.21.0.0/16", "controller_version": "1.2.3", "nethserver_version": "8-23.05.3-ns.1.0.1", "nethserver_system_id": "XXXXXXXX-XXXX", "metrics_retention_days": 60, "logs_retention_days": 30 }, "message": "success" }
Basic authentication API
-
GET
/authThis endpoint is used to check if the user is authenticated. It returns a 200 status code if the user is authenticated, otherwise it returns a 401 status code. It can be used by external applications to check if the user is authenticated without needing to handle JWT tokens.
REQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 X-Auth-User: <username> { "authentication": "ok" }
Defaults
-
GET /defaultsREQ
Content-Type: application/json Authorization: Bearer <JWT_TOKEN>RES
HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 { "code": 200, "data": { "fqdn": "controller.ns8.local", "grafana_path": "/grafana", "prometheus_path": "/prometheus", "webssh_path": "/webssh", "valid_subscription": false }, "message": "success" }
Ingest
This API is used to ingest metrics from connected units. It requires basic authentication and
takes firewall_api as a parameter.
The firewall_api paramater is the name of the firewall API that is sending the metrics.
The API accepts only POST requests abd requires the following headers:
Authorization:: basic authentication header, where the username is the unit uuid and the password is the registration tokenContent-Type: application/json: the content type must be JSON
It responds with a 200 status code in case of success. Success example:
{ "code": 200, "data": null, "message": "success" }
Possible error status codes are:
- 400 if the request is malformed
- 401 if the authentication headers are missing or invalid
- 500 if there is an internal server error
Error example:
{ "code": 401, "data": null, "message": "invalid unit id" }
-
POST /ingest/dump-nsplug-configCreate the unit record where all metrics are connected, it also stores the unit name in the report database. This endpoint is mandatory and must be called at least once before sending all other metrics.
REQ
{ "name": "fw.test.local" } -
POST /ingest/dump-mwan-eventsStore all multiwan events in the report database.
REQ
{ "data": [ { "timestamp": 1726819981, "wan": "wan", "interface": "eth1", "event": "online" }, { "timestamp": 1726820241, "wan": "wan2", "interface": "eth2", "event": "offline" } ] } -
POST /ingest/dump-ts-attacksStore all threat shield brute force blocks (fail2ban-like) in the report database.
REQ
{ "data": [ { "timestamp": 1726812650, "ip": "200.91.234.36" } ] } -
POST /ingest/dump-ts-malwareStore all threat shield blocks based on category in the report database.
REQ
{ "data": [ { "timestamp": 1726811160, "src": "5.6.32.54", "dst": "1.2.3.4", "category": "nethesislvl3v4", "chain": "inp-wan" } ] } -
POST /ingest/dump-ovpn-connectionsStore all openvpn connections in the report database.
REQ
{ "data": [ { "timestamp": 1726812276, "instance": "ns_roadwarrior1", "common_name": "user1", "virtual_ip_addr": "10.9.10.41", "remote_ip_addr": "1.2.3.4", "start_time": 1726819476, "duration": 4, "bytes_received": 16343, "bytes_sent": 7666 } ] } -
POST /ingest/dump-dpi-statsStore all network traffic stats in the report database.
REQ
{ "data": [ { "timestamp": 1726819203, "client_address": "fe80::10ac:f709:5fb8:8fc3", "client_name": "host1.test.local", "protocol": "mdns", "bytes": 123 } ] } -
POST /ingest/dump-ovpn-configStore the openvpn configuration in the report database.
REQ
{ "data": [ { "instance": "ns_roadwarrior1", "device": "tunrw1", "type": "rw", "name": "srv1" } ] } -
POST /ingest/dump-wan-configREQ
{ "data": [ { "interface": "wan1", "device": "eth0", "status": "online" }, { "interface": "wan2", "device": "eth5", "status": "offline" } ] } -
POST /ingest/infoThis endpoint is used to store general information about the unit in the report database. It requires basic authentication and accepts the following headers:
Authorization:: basic authentication header, where the username is the unit UUID and the password is the registration token.Content-Type: application/json: the content type must be JSON.
REQ
{ "unit_name": "NethSec", "version": "NethSecurity 8 24.10.0-ns.1.6.0", "subscription_type": "", "system_id": "", "ssh_port": 22, "fqdn": "NethSec", "description": "This is my description", "api_version": "3.2.0-r1", "scheduled_update": -1, "version_update": "NethSecurity 8-24.10.0-ns.1.6.0" }RES
{ "code": 200, "data": null, "message": "success" }Possible error status codes:
- 400 if the request is malformed.
- 401 if the authentication headers are missing or invalid.
- 500 if there is an internal server error.
Error example:
{ "code": 401, "data": null, "message": "invalid unit id" }
Testing
Execute tests with coverage:
go test ./... -coverpkg=./... -coverprofile=coverage.out -v
Documentation
¶
There is no documentation for this package.