Upnode Security Architecture

Team’s experience and track record

We are a team of experienced node operators and blockchain developers with over 30 years of combined experience in Web 2.0 and Web 3.0 deployments. We have a track record in the development of and contributions to multiple tools to support our node operations. We were awarded the Optimism Grant in Cycle 14 to build Upnode Deploy, an all-in-one API Deployment Tool for web3 developers. Please see our proposal.

Infrastructure and Security setup

Upnode employs an Active-Standby validator system across multiple regions for maximum availability, protected by a VPN network managed by a Fortigate firewall. Servers maintain advanced firewalls with limited port exposure to prevent unattended access. Private keys are stored securely using YubiHSM2; if not supported, we use YubiKey for the SSH key of the validator node. We employ two different OS for each of our datacenters, one relies on Proxmox VMs for node isolation, while the other relies on Ubuntu bare metal for optimal performance, which is our primary active node. We have an intensive monitoring and alerting stack with Tenderduty, Grafana, Loki, Prometheus Alertmanager, and more with alerts directed to our engineer 24/7 through PagerDuty for incident management. Please see the in-depth detail below

Upnode prioritizes server security through a layered approach. An IP whitelist is implemented to ensure that only trusted IP addresses can access the system. User permissions are set up to ensure that each operator has only the necessary rights to perform their tasks. Above all, the entire infrastructure is guarded under the Fortigate 200F hardware firewall VPN. This means any access attempt, regardless of the method, must first authenticate through this fortified VPN barrier, adding an additional layer of defense against intrusions.

We have two engineers working in different timezones. When an incident occurs, both will be on-call. Whoever is working at the time will address the issue 24/7.

We have two servers, each equipped with 2× AMD EPYC 75F3 32-core processors and 640 GB of RAM. We only use NVMe drives and with expandable slots capable of supporting more than 100 TB of additional NVMe storage.

Upnode uses an Active-Standby validator setup across various geographic regions for disaster recovery. Access to the validator infrastructure is via a VPN network through an enterprise-grade Fortigate 200F hardware firewall. All servers run advanced firewalls and only essential ports are exposed.

We use YubiHSM2 hardware signer to securely store private keys and sign the validated block if supported by the validator software of that chain. However, if YubiHSM2 is not officially supported yet, we use YubiKey to store the SSH key of our validator node and restrict access to our validator server to only a few operators.

Datacenter 1 and Datacenter 2 operate on distinct architectures. Datacenter 1 utilizes nodes within Proxmox VMs, offering a notable advantage in node isolation. In contrast, Datacenter 2 employs Ubuntu bare metal, optimized for high-performance and low-latency tasks.

By default, we rely on our Ubuntu bare metal as the primary active node due to its superior performance and minimal latency. However, should any issues arise that compromise the Ubuntu system, we have a backup server. This backup is safeguarded by the robust node isolation provided by virtualization, reducing the likelihood of concurrent system crashes.

We have implemented a robust and comprehensive monitoring stack, incorporating Grafana, Loki, and Prometheus Alertmanager. This powerful combination allows us to effectively monitor the health and performance of your infrastructure. In the event of any incidents or anomalies, alerts are promptly generated and sent to PagerDuty, which serves as a centralized incident management platform.

We utilize ZFS encrypted pool technology to store and encrypt our data to be secured from physical attack for nodes that aren’t support YubiHSM