You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+99-21Lines changed: 99 additions & 21 deletions
Original file line number
Diff line number
Diff line change
@@ -6,21 +6,23 @@ There are a couple x86 boards out there. The closest to the UDDO in price and fe
6
6
7
7
I wanted to build a case around all the components. A friend prototyped an acrylic case that could be laser cut. I immediately started playing with Onshape to take the idea to completion. After learning how to sketch and build the panels, I took the drawings to Pololu for supplying and cutting the acrylic.
8
8
9
-
#Build
9
+
## Components
10
10
11
11
Let's go through some of the main build components and how they came to be. Some of these could be swapped out for different sizes or models. I'll try and point out some of the places where I would have done differently.
12
12
13
13
### Boards
14
14
15
+
The main board is the UDOO X86 Advanced.
16
+
15
17
I did some research on boards. I had originally wanted to go with a Raspberry Pi design. But like most of the other ARM platforms I found in the Pi price range, I just ran into too many limitations for what I wanted to do (the biggest one being lack of proper PXE support). The ARM compatibilities around containers was also frustrating enough that I wanted to go x86_64. Though, I imagine in another year or two, we'll be able to have proper LAN boot options and more transparent multi-architecture support in technologies like Docker.
16
18
17
19
The UDOO X86 Advanced was the board I settled on. Some of the stuff I wanted to work with (such as Juju) require 4GB of RAM. The Advanced version of the board fit the bill. I imagine two ways to cut costs would be to either go with the Advanced Plus model (which has 32GB eMMC storage built-in) or go with the Basic model and use a cheaper SD card as storage and get only 2GB RAM (in my notes near the bottom I talk about SD storage and why I chose to use a M.2 SSD instead).
18
20
19
21
### Case
20
22
21
-
The case was the most fun exercise. I had never used any CAD style tooling before so there was a definite learning curve. But I had some help from a friend who had experience. I stumbled upon Onshape and it felt like a very polished tool. I am actually surprised at how much it is able to do in the browser.
23
+
The case is built from laser cut 3mm thick acrylic panels. The CAD is actually public at https://cad.onshape.com/documents/991c4a5831e2f997a4a71842/w/bb91f76a25a96fa561acf156/e/013c2a1a40829f74b89b5690. Anyone can open it up and check out the parts and assembly!
22
24
23
-
The document is actually public at https://cad.onshape.com/documents/991c4a5831e2f997a4a71842/w/bb91f76a25a96fa561acf156/e/013c2a1a40829f74b89b5690. Anyone can open it up and check out the parts and assembly!
25
+
The case was the most fun exercise. I had never used any CAD style tooling before so there was a definite learning curve. But I had some help from a friend who had experience. I stumbled upon Onshape and it felt like a very polished tool. I am actually surprised at how much it is able to do in the browser.
24
26
25
27
I did not have any calipers (I do now!) for measuring parts at the time. Most of my part and mounting dimensions were from ruler and/or visual only. This led to issues that popped up during assembly:
26
28
@@ -29,35 +31,110 @@ I did not have any calipers (I do now!) for measuring parts at the time. Most of
29
31
* The notches in the top of the rear panel are not lined up perfectly with the notches in the top panel (off by about 1mm). I was able to use some wire cutters to try the notch in the top panel so it would fit. You can actually see this in the Onshape assembly, I just missed it during my review.
30
32
* The bottom panel bears most of the weight yet it only attaches to the sides. This could probably be designed a little better to attach at the front and back as well. Though, being able to screw the top panel into the standoffs help alleviate any pressure by the boards. Not as bad as it sounds (you can't really see it bowing) but it could be better.
31
33
* The edges of the case have some flex to them. The notches could have probably been spread out more evenly instead of right next to the bolt holes.
32
-
* The front air vents are off by 1.6mm compounding for each vent as I forgot to account for the width of the board. Oops again!
34
+
* The front air vents are off by 1.6mm compounding for each vent as I forgot to account for the thickness of the board. Oops again!
35
+
36
+
The case I built was version `V3`. I went through and tweaked a bunch of constraints for the some of the issues above. If I ever build another case (or if you do, please let me know) I will make a version `V4`.
37
+
38
+
### Power Supply
39
+
40
+
The power supply unit (PSU) is a Mean Well RS-100-12. I would later learn that this is an older model and has been replaced by the LRS-100-12. The LRS-100-12 is smaller, has a better efficiency rating (should run cooler), and slightly cheaper. For new builds I'd recommend just using the LRS-100-12.
41
+
42
+
I looked at dual voltage power supplies (since I was initially thinking I would run a 5v switch and that the fan would be 5v as well). But it turned out to be much bulkier to get the wattage needed at 12 and 5 volts. I also finally found a 12v based switch and realized that most computer fans are actually 12v.
43
+
44
+
### Networking
45
+
46
+
The networking components consist of a Netgear GS108 switch, MikroTik hEX router, and some Monoprice SlimRun ethernet cables. The hEX has mechanical drawings available but I couldn't find any for the GS108. Fortunately, it had mounting holes on the board and I was (mostly) able to figure out the correct hole to hole dimensions.
47
+
48
+
The GS108 was the only switch ii found that was powered by 12 volts (most others were 5 or 9). The hEX has a very wide range input voltage so no worries there. But both the GS108 and hEX are a bit pricey for what they are doing. I think the hEX could be swapped out for the hEX lite and the GS108 for another 12v switch if I searched longer. I am also not fond of the GS108 DC power jack on the back. Had it been on the same side of the ethernet ports I would have been able to move the switch forward a bit more to create some cable space.
49
+
50
+
The hEX acts not only as a NAT gateway for the internal network but also as a VPN server. To get into the network I create a L2TP VPN connection in my macOS network settings. Once connected, I get access as if I was directly connected to the switch.
51
+
52
+
I go into the router and VPN configuration specifics later in the installation instructions.
53
+
54
+
### Fan, Ethernert, C14
55
+
56
+
The fan is a Antec TwoCool 140mm. The ethernet jack and C13 receptacle were just whatever I found available on Amazon. I'm sure any comparable parts would work as long as the dimensions were tweaked to fit them.
57
+
58
+
I got the fan in a dual speed model because I wasn't sure how much airflow I needed. So far, the low speed on the fan works well and seems to be keeping everything cool. I haven't measured temperatures yet. But just touching the PSU and heatsinks on the boards feels much cooler than being in the open air.
59
+
60
+
### Other
61
+
62
+
There are some other various parts that I already had on hand but I'll try to list them out in case they are needed:
33
63
34
-
## Setup
64
+
* M3 x 5mm nylon standoffs (brass or metal would be fine as well).
65
+
* M3 washers (for spacing the boards off the base panel)
66
+
* 18 AWG wire (for building the wiring harness from the PSU to the DC pigtails)
67
+
* Various tools (philips screw driver, wire cutters, wire tap crimpers)
68
+
* Soldering iron and solder
69
+
* Spade terminals for 18 AWG wire (for easy connecting to C14 receptacle. though, you could also just solder the wires on).
70
+
* Eye hole terminals for 18 AWG wire to PSU outputs and inputs (though, you could probably just screw the bare wires in if you wanted).
71
+
* Shrink tubing in various sizes.
72
+
* Double sided foam tape.
35
73
36
-
### UDOO BIOS
74
+
##Instructions
37
75
38
-
* Set boot type to legacy
39
-
* Enable PXE boot to LAN
40
-
* Under Legacy submenu navigate to boot type order.
41
-
* Use the - key to move the hard drive under _Other_ (PXE boot).
76
+
Now that you have all the parts, let's get building!
42
77
43
-
### Head Node
78
+
### Router Configuration
44
79
45
-
#### Install MAAS
80
+
Before we install the router we need to do some pre-configuration so we can connect to it.
81
+
82
+
1. Connect up your computer to the second ethernet port and make sure it is configured for DHCP.
83
+
1. Power up the hEX with the included power adapter.
84
+
1. SSH or telnet to 192.168.88.1.
85
+
1. Copy and paste the configuration from https://github.com/andyshinn/kuberdoo/blob/master/config/default_vpn.rsc (this should get the NAT, firewall, VPN, and basics set up).
86
+
87
+
After loading the configuration, connect port 1 up to your home network, power cycle the hEX, and verify that you can connect to it (you will need to consult your home router to find out what address the MikroTik was given). For example, my home network is 10.0.1.0/24 and the MikroTik got an address of 10.0.1.58. I am able to connect to the device via SSH at 10.0.1.58 with user `admin` and password `kub3rd00`. There is also a web interface at http://10.0.1.58/ if you prefer to configure the device that way.
88
+
89
+
Now that we have verified the device can be remotely connected to, it is ready to be installed into the case. You can leave the router powered up because we'll use the TFTP server configured on it to bootstrap the first UDOO board next.
90
+
91
+
### UDOO Preparation
92
+
93
+
In addition to the router, we need to prep the UDOO boards so that they properly boot from PXE and failover to hard drive when no PXE server is available. We also need to install Ubuntu on one of the boards so that we have a node to bootstrap from.
94
+
95
+
1. Install the M.2 SSDs onto all the boards. The UDOO X86 should have come with a small bag of M.2 mounting hardware.
96
+
1. Use a small piece of double sided foam tape to hold down the CMOS battery.
97
+
1. Connect up a UDOO to monitor via HDMI or mini DisplayPort, keyboard, and a 12 volt DC power adapter. You can use the one that comes with the Netgear GS108 switch. But don't use the adapter that comes with the MikroTik hEX as it is 24 volts!
98
+
1. Start repeatedly pressing the <kbd>ESC</kbd> key on the keyboard while applying power to the UDOO board.
99
+
1. Enter the SCU once at the setup screen.
100
+
1. Navigate to _Power_ and set the _Power Fail Resume Type_ to _Always OFF_ (we don't want to boards automatically powering on).
101
+
1. Navigate to _Boot_ menu.
102
+
1. Set _Boot Type_ to _Legacy_ (UEFI doesn't failover to hard disk properly when PXE fails).
103
+
1. Enable _PXE boot to LAN_.
104
+
1. Navigate to _Exit_ and choose _Exit Saving Changes_.
105
+
1. Keep pressing the ESC key as the device reboots again.
106
+
1. Navigate to the _Boot_ menu again and there should be a new _Legacy_ menu.
107
+
1. Under _Legacy_ submenu navigate to _Boot Type Order_.
108
+
1. Highlight _Hard Disk Drive_ and use the <kbd>-</kbd> key to move it under _Other_ (so that PXE will boot before the SSD).
109
+
1. Navigate to _Exit_ and choose _Exit Saving Changes_.
110
+
111
+
Repeat these steps for all the boards. It is difficult to hook up HDMI and keyboard to the boards while they are in the case so this is why we do it now. We want the boards to boot and install headless.
112
+
113
+
Choose one of the boards to be the head node. This is the board we'll use to install Ubuntu onto and will run Ubuntu MAAS or other PXE server to bootstrap the other boards. With this board:
114
+
115
+
1. Connect the board up to your monitor and keyboard again.
116
+
1. Connect this board to one of the ethernet ports on the MikroTik hEX.
117
+
1. Power on the board and it should PXE boot into https://netboot.xyz/ from the TFTP server we previously set up on the hEX.
118
+
1. Navigate the menus to install Ubuntu 16.04 or other flavor of Linux (if you plan to use something other than MAAS).
119
+
1. Follow through the installation and verify you can SSH to the node afterwards.
120
+
121
+
Once we can remotely connect to this node it is safe to go in the case. We know that we can connect to it without needing monitor and keyboard. It may be helpful to number the boards at this point. I put a number 1 on my head node CMOS battery and labeled the others boards 2 through 6.
122
+
123
+
### Wiring Harness
124
+
125
+
I started by mocking up the boards on the bottom panel and the PSU on the side panel. This helped give an idea of how long the DC plug pigtails needed to be for the wiring harness. I ended up making two, one harness for the 6 boards and one for the fan, switch, and router. The harnesses are the pigtails soldered to 18 AWG wire leads that plug into the positive and negative terminals on the PSU.
126
+
127
+
### Mounting Switch
128
+
129
+
### Mounting Router
130
+
131
+
### Head Node MAAS
46
132
47
133
```
48
134
sudo apt install maas
49
135
sudo maas createadmin
50
136
```
51
137
52
-
### Network
53
-
54
-
The MikroTik hEX has a default IP address of 192.168.88.1 and offers DHCP on ports 2-5. Configure your computer to get DHCP from the ethernet port, connect it up to the hEX on port 2, and then navigate to http://192.168.88.1/. Here, we'll be able to configure a password and VPN settings.
55
-
56
-
* Disable DHCP.
57
-
* Change password.
58
-
* Add L2TP server.
59
-
* Add UDP ports 500,4500 to firewall for L2TP.
60
-
* Add DNS server 192.168.88.11.
61
138
62
139
## Notes
63
140
@@ -90,3 +167,4 @@ Some other thoughts on other physical cluster stuff I'd love to replicate on the
90
167
* Bonding and port failover. Mikrotik handles this. But we'd need boards that had multiple interfaces (UP squared might be a good candidate for this).
91
168
* VLANs and routing between them. Mikrotik already supports this. Would t least need a switch that could also handle VLANs and tagging / trunking.
92
169
* IPMI / OOB management. The onboard Arduino has the ability to power on/off the main board. I wonder if it could act as a simple IPMI device?
170
+
* HDMI / USB case ports for each board. It should be possible to widen the case and make room for some HDMI and USB passthrough cables for each board. This would make it easier to service (changing BIOS options, debugging broken boot, etc.)
0 commit comments