The purpose of this documentation is to simplify local SingleStore Cluster in a box setup in Windows using WSL2, Ubuntu for WSL2 and Docker Desktop for WSL backend and KinD (Kubernetes in Docker). Read on to know more.
The Windows Subsystem for Linux lets developers run a GNU/Linux environment -- including most command-line tools, utilities, and applications -- directly on Windows, unmodified, without the overhead of a traditional virtual machine or dualboot setup.
You can:
- Choose your favorite GNU/Linux distributions from the Microsoft Store.
- Run common command-line tools such as
grep,sed,awk, or other ELF-64 binaries. - Run Bash shell scripts and GNU/Linux command-line applications including:
- Tools: vim, emacs, tmux
- Languages: NodeJS, Javascript, Python, Ruby, C/C++, C# & F#, Rust, Go, etc.
- Services: SSHD, MySQL, Apache, lighttpd, MongoDB, PostgreSQL.
- Install additional software using your own GNU/Linux distribution package manager.
- Invoke Windows applications using a Unix-like command-line shell.
- Invoke GNU/Linux applications on Windows.
[!div class="nextstepaction"] Install WSL
WSL 2 is a new version of the Windows Subsystem for Linux architecture that powers the Windows Subsystem for Linux to run ELF64 Linux binaries on Windows. Its primary goals are to increase file system performance, as well as adding full system call compatibility.
This new architecture changes how these Linux binaries interact with Windows and your computer's hardware, but still provides the same user experience as in WSL 1 (the current widely available version).
Individual Linux distributions can be run with either the WSL 1 or WSL 2 architecture. Each distribution can be upgraded or downgraded at any time and you can run WSL 1 and WSL 2 distributions side by side. WSL 2 uses an entirely new architecture that benefits from running a real Linux kernel.
For simplicity, we generally recommend using the wsl --install to install Windows Subsystem for Linux, but if you're running an older build of Windows, that may not be supported. We have included the manual installation steps below. If you run into an issue during the install process, check the installation section of the troubleshooting guide.
You must first enable the "Windows Subsystem for Linux" optional feature before installing any Linux distributions on Windows.
Open PowerShell as Administrator (Start menu > PowerShell > right-click > Run as Administrator) and enter this command:
dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestartWe recommend now moving on to step #2, updating to WSL 2, but if you wish to only install WSL 1, you can now restart your machine and move on to Step 6 - Install your Linux distribution of choice. To update to WSL 2, wait to restart your machine and move on to the next step.
To update to WSL 2, you must be running Windows 10.
- For x64 systems: Version 1903 or higher, with Build 18362 or higher.
- For ARM64 systems: Version 2004 or higher, with Build 19041 or higher.
- Builds lower than 18362 do not support WSL 2. Use the Windows Update Assistant to update your version of Windows.
To check your version and build number, select Windows logo key + R, type winver, select OK. Update to the latest Windows version in the Settings menu.
Note
If you are running Windows 10 version 1903 or 1909, open "Settings" from your Windows menu, navigate to "Update & Security" and select "Check for Updates". Your Build number must be 18362.1049+ or 18363.1049+, with the minor build # over .1049. Read more: WSL 2 Support is coming to Windows 10 Versions 1903 and 1909.
Before installing WSL 2, you must enable the Virtual Machine Platform optional feature. Your machine will require virtualization capabilities to use this feature.
Open PowerShell as Administrator and run:
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestartRestart your machine to complete the WSL install and update to WSL 2.
-
Download the latest package:
[!NOTE] If you're using an ARM64 machine, please download the ARM64 package instead. If you're not sure what kind of machine you have, open Command Prompt or PowerShell and enter:
systeminfo | find "System Type". Caveat: On non-English Windows versions, you might have to modify the search text, translating the "System Type" string. You may also need to escape the quotations for the find command. For example, in Germansysteminfo | find '"Systemtyp"'. -
Run the update package downloaded in the previous step. (Double-click to run - you will be prompted for elevated permissions, select ‘yes’ to approve this installation.)
Once the installation is complete, move on to the next step - setting WSL 2 as your default version when installing new Linux distributions. (Skip this step if you want your new Linux installs to be set to WSL 1).
Note
For more information, read the article changes to updating the WSL2 Linux kernel, available on the Windows Command Line Blog.
Open PowerShell and run this command to set WSL 2 as the default version when installing a new Linux distribution:
wsl --set-default-version 2-
Open the Microsoft Store and select your favorite Linux distribution.
The following links will open the Microsoft store page for each distribution:
-
From the distribution's page, select "Get".
The first time you launch a newly installed Linux distribution, a console window will open and you'll be asked to wait for a minute or two for files to de-compress and be stored on your PC. All future launches should take less than a second.
You will then need to create a user account and password for your new Linux distribution.
CONGRATULATIONS! You've successfully installed and set up a Linux distribution that is completely integrated with your Windows operating system!
If you run into an issue during the install process, check the installation section of the troubleshooting guide.
There are some scenarios in which you may not be able (or want) to, install WSL Linux distributions using the Microsoft Store. You may be running a Windows Server or Long-Term Servicing (LTSC) desktop OS SKU that doesn't support Microsoft Store, or your corporate network policies and/or admins do not permit Microsoft Store usage in your environment. In these cases, while WSL itself is available, you may need to download Linux distributions directly.
If the Microsoft Store app is not available, you can download and manually install Linux distributions using these links:
- Ubuntu
- Ubuntu 20.04
- Ubuntu 20.04 ARM
- Ubuntu 18.04
- Ubuntu 18.04 ARM
- Ubuntu 16.04
- Debian GNU/Linux
- Kali Linux
- SUSE Linux Enterprise Server 12
- SUSE Linux Enterprise Server 15 SP2
- SUSE Linux Enterprise Server 15 SP3
- openSUSE Tumbleweed
- openSUSE Leap 15.3
- openSUSE Leap 15.2
- Oracle Linux 8.5
- Oracle Linux 7.9
- Fedora Remix for WSL
This will cause the <distro>.appx packages to download to a folder of your choosing.
If you prefer, you can also download your preferred distribution(s) via the command line, you can use PowerShell with the Invoke-WebRequest cmdlet. For example, to download Ubuntu 20.04:
Invoke-WebRequest -Uri https://aka.ms/wslubuntu2004 -OutFile Ubuntu.appx -UseBasicParsingTip
If the download is taking a long time, turn off the progress bar by setting $ProgressPreference = 'SilentlyContinue'
You also have the option to use the curl command-line utility for downloading. To download Ubuntu 20.04 with curl:
curl.exe -L -o ubuntu-2004.appx https://aka.ms/wslubuntu2004In this example, curl.exe is executed (not just curl) to ensure that, in PowerShell, the real curl executable is invoked, not the PowerShell curl alias for Invoke-WebRequest.
Once the distribution has been downloaded, navigate to the folder containing the download and run the following command in that directory, where app-name is the name of the Linux distribution .appx file.
Add-AppxPackage .\app_name.appxOnce the Appx package has finished downloading, you can start running the new distribution by double-clicking the appx file. (The command wsl -l will not show that the distribution is installed until this step is complete).
If you are using Windows server, or run into problems running the command above you can find the alternate install instructions on the Windows Server documentation page to install the .appx file by changing it to a zip file.
Once your distribution is installed, follow the instructions to create a user account and password for your new Linux distribution.
Using Windows Terminal enables you to open multiple tabs or window panes to display and quickly switch between multiple Linux distributions or other command lines (PowerShell, Command Prompt, Azure CLI, etc). You can fully customize your terminal with unique color schemes, font styles, sizes, background images, and custom keyboard shortcuts. Learn more.
Once WSL 2 installed, we could proceed to install Docker Desktop for Windows and enable WSL2 backend.
Update to the Docker Desktop terms
Commercial use of Docker Desktop in larger enterprises (more than 250 employees OR more than $10 million USD in annual revenue) now requires a paid subscription. The grace period for those that will require a paid subscription ends on January 31, 2022. Learn more{: target="blank" rel="noopener" class="" id="dkr_docs_cta"}. {: .important}
Windows Subsystem for Linux (WSL) 2 introduces a significant architectural change as it is a full Linux kernel built by Microsoft, allowing Linux containers to run natively without emulation. With Docker Desktop running on WSL 2, users can leverage Linux workspaces and avoid having to maintain both Linux and Windows build scripts. In addition, WSL 2 provides improvements to file system sharing, boot time, and allows access to some cool new features for Docker Desktop users.
Docker Desktop uses the dynamic memory allocation feature in WSL 2 to greatly improve the resource consumption. This means, Docker Desktop only uses the required amount of CPU and memory resources it needs, while enabling CPU and memory-intensive tasks such as building a container to run much faster.
Additionally, with WSL 2, the time required to start a Docker daemon after a cold start is significantly faster. It takes less than 10 seconds to start the Docker daemon when compared to almost a minute in the previous version of Docker Desktop.
Before you install the Docker Desktop WSL 2 backend, you must complete the following steps:
- Install Windows 10, version 1903 or higher or Windows 11.
- Enable WSL 2 feature on Windows. For detailed instructions, refer to the Microsoft documentation{:target="blank" rel="noopener" class=""}.
- Download and install the Linux kernel update package{:target="blank" rel="noopener" class=""}.
-
To get the best out of the file system performance when bind-mounting files, we recommend storing source code and other data that is bind-mounted into Linux containers (i.e., with
docker run -v <host-path>:<container-path>) in the Linux file system, rather than the Windows file system. You can also refer to the recommendation{:target="blank" rel="noopener" class=""} from Microsoft.- Linux containers only receive file change events ("inotify events") if the original files are stored in the Linux filesystem. For example, some web development workflows rely on inotify events for automatic reloading when files have changed.
- Performance is much higher when files are bind-mounted from the Linux
filesystem, rather than remoted from the Windows host. Therefore avoid
docker run -v /mnt/c/users:/users(where/mnt/cis mounted from Windows). - Instead, from a Linux shell use a command like
docker run -v ~/my-project:/sources <my-image>where~is expanded by the Linux shell to$HOME.
-
If you have concerns about the size of the docker-desktop-data VHDX, or need to change it, take a look at the WSL tooling built into Windows{:target="blank" rel="noopener" class=""}.
-
If you have concerns about CPU or memory usage, you can configure limits on the memory, CPU, Swap size allocated to the WSL 2 utility VM{:target="blank" rel="noopener" class=""}.
-
To avoid any potential conflicts with using WSL 2 on Docker Desktop, you must uninstall any previous versions of Docker Engine and CLI installed directly through Linux distributions before installing Docker Desktop.
Download Docker Desktop 2.3.0.2 or a later release.
Ensure you have completed the steps described in the Prerequisites section before installing the Docker Desktop Stable 2.3.0.2 release.
-
Follow the usual installation instructions to install Docker Desktop. If you are running a supported system, Docker Desktop prompts you to enable WSL 2 during installation. Read the information displayed on the screen and enable WSL 2 to continue.
-
Start Docker Desktop from the Windows Start menu.
-
From the Docker menu, select Settings > General.
-
Select the Use WSL 2 based engine check box.
If you have installed Docker Desktop on a system that supports WSL 2, this option will be enabled by default.
-
Click Apply & Restart.
-
Ensure the distribution runs in WSL 2 mode. WSL can run distributions in both v1 or v2 mode.
To check the WSL mode, run:
wsl.exe -l -vTo upgrade your existing Linux distro to v2, run:
wsl.exe --set-version (distro name) 2To set v2 as the default version for future installations, run:
wsl.exe --set-default-version 2 -
When Docker Desktop restarts, go to Settings > Resources > WSL Integration.
The Docker-WSL integration will be enabled on your default WSL distribution. To change your default WSL distro, run
wsl --set-default <distro name>.For example, to set Ubuntu as your default WSL distro, run
wsl --set-default ubuntu.Optionally, select any additional distributions you would like to enable the Docker-WSL integration on.
Note
The Docker-WSL integration components running in your distro depend on glibc. This can cause issues when running musl-based distros such as Alpine Linux. Alpine users can use the alpine-pkg-glibc{:target="blank" rel="noopener" class=""} package to deploy glibc alongside musl to run the integration.
-
Click Apply & Restart.
The following section describes how to start developing your applications using Docker and WSL 2. We recommend that you have your code in your default Linux distribution for the best development experience using Docker and WSL 2. After you have enabled WSL 2 on Docker Desktop, you can start working with your code inside the Linux distro and ideally with your IDE still in Windows. This workflow can be pretty straightforward if you are using VSCode.
-
Open VSCode and install the Remote - WSL extension. This extension allows you to work with a remote server in the Linux distro and your IDE client still on Windows.
-
Now, you can start working in VSCode remotely. To do this, open your terminal and type:
wslcode .This opens a new VSCode connected remotely to your default Linux distro which you can check in the bottom corner of the screen.
Alternatively, you can type the name of your default Linux distro in your Start menu, open it, and then run
code. -
When you are in VSCode, you can use the terminal in VSCode to pull your code and start working natively from your Windows machine.
Starting with Docker Desktop 3.1.0, Docker Desktop supports WSL 2 GPU Paravirtualization (GPU-PV) on NVIDIA GPUs. To enable WSL 2 GPU Paravirtualization, you need:
- A machine with an NVIDIA GPU
- The latest Windows Insider version from the Dev Preview ring
- Beta drivers{:target="blank" rel="noopener" class=""} from NVIDIA supporting WSL 2 GPU Paravirtualization
- Update WSL 2 Linux kernel to the latest version using
wsl --updatefrom an elevated command prompt - Make sure the WSL 2 backend is enabled in Docker Desktop
To validate that everything works as expected, run the following command to run a short benchmark on your GPU:
$ docker run --rm -it --gpus=all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark
Run "nbody -benchmark [-numbodies=<numBodies>]" to measure performance.
-fullscreen (run n-body simulation in fullscreen mode)
-fp64 (use double precision floating point values for simulation)
-hostmem (stores simulation data in host memory)
-benchmark (run benchmark to measure performance)
-numbodies=<N> (number of bodies (>= 1) to run in simulation)
-device=<d> (where d=0,1,2.... for the CUDA device to use)
-numdevices=<i> (where i=(number of CUDA devices > 0) to use for simulation)
-compare (compares simulation results running once on the default GPU and once on the CPU)
-cpu (run n-body simulation on the CPU)
-tipsy=<file.bin> (load a tipsy model file for simulation)
> NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
> Windowed mode
> Simulation data stored in video memory
> Single precision floating point simulation
> 1 Devices used for simulation
MapSMtoCores for SM 7.5 is undefined. Default to use 64 Cores/SM
GPU Device 0: "GeForce RTX 2060 with Max-Q Design" with compute capability 7.5
> Compute 7.5 CUDA device: [GeForce RTX 2060 with Max-Q Design]
30720 bodies, total time for 10 iterations: 69.280 ms
= 136.219 billion interactions per second
= 2724.379 single-precision GFLOP/s at 20 flops per interactionNow that we have WSL2 and Docker Destop installed, you could enable KinD (Kubernetes in Docker)
Docker Desktop includes a standalone Kubernetes server and client, as well as Docker CLI integration that runs on your machine. The Kubernetes server runs locally within your Docker instance, is not configurable, and is a single-node cluster.
The Kubernetes server runs within a Docker container on your local system, and is only for local testing. Enabling Kubernetes allows you to deploy your workloads in parallel, on Kubernetes, Swarm, and as standalone containers. Enabling or disabling the Kubernetes server does not affect your other workloads.
The Kubernetes client command kubectl is included and configured to connect
to the local Kubernetes server. If you have already installed kubectl and
pointing to some other environment, such as minikube or a GKE cluster, ensure you change the context so that kubectl is pointing to docker-desktop:
$ kubectl config get-contexts
$ kubectl config use-context docker-desktopIf you installed kubectl using Homebrew, or by some other method, and
experience conflicts, remove /usr/local/bin/kubectl.
To enable Kubernetes support and install a standalone instance of Kubernetes running as a Docker container, go to Preferences > Kubernetes and then click Enable Kubernetes.
By default, Kubernetes containers are hidden from commands like docker service ls, because managing them manually is not supported. To see these internal containers, select Show system containers (advanced). Most users do not need this option.
Click Apply & Restart to save the settings and then click Install to confirm. This instantiates images required to run the Kubernetes server as containers, and installs the /usr/local/bin/kubectl command on your machine.
When Kubernetes is enabled and running, an additional status bar item displays at the bottom right of the Docker Desktop Settings dialog.
The status of Kubernetes shows in the Docker menu and the context points to docker-desktop.
Upgrade Kubernetes
Docker Desktop does not upgrade your Kubernetes cluster automatically after a new update. To upgrade your Kubernetes cluster to the latest version, select Reset Kubernetes Cluster.
Kubernetes integration provides the Kubernetes CLI command
at /usr/local/bin/kubectl on Mac and at C:\>Program Files\Docker\Docker\Resources\bin\kubectl.exe on Windows. This location may not be in your shell's PATH
variable, so you may need to type the full path of the command or add it to
the PATH.
You can test the command by listing the available nodes:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
docker-desktop Ready master 3h v1.19.7For more information about kubectl, see the
kubectl documentation{:target="blank" rel="noopener" class=""}.
To disable Kubernetes support at any time, clear the Enable Kubernetes check box. This stops and removes Kubernetes containers, and also removes the /usr/local/bin/kubectl command.
Once we have Kubernetes up and running with KinD, we can now deploy our Single Store Cluster-In-A-Box
Even though SingleStore is a distributed system, we could run a minimal version of SingleStore on your laptop in Kubernetes. The combination of free access, and being able to run SingleStore on your laptop, can be extremely convenient for demos, software testing, developer productivity. Read on to find out how.
A machine with at least 8GB RAM and four CPUs is recommended for this setup. This is ideal for quickly provisioning a system to understand the capabilities of the SQL engine. Kubernetes deployments are self sustained. Once deployed, except for customization, there would no further overhead of managing or configuring SingleStore to get it running.
 Typically each yaml file contains a single resource. For simplicity, we’ll create one yaml file that includes both a deployment and a service. We’ll connect to the service, the service will proxy to the pod, and the pod will route the request into the container.
We’ll use the memsql/cluster-in-a-box image built by SingleStore and available on Docker Hub. This image comes with the SingleStore database engine and SingleStore Studio preinstalled. The minimum system requirements are disabled in this “cluster-in-a-box” configuration.
Create an empty directory, and create a file named kubernetes-memsql.yaml inside. Open this file in your favorite code editor and paste in this content. As with Python source code, white space is significant.
Yaml uses two spaces, not tabs. Ensure each section is indented with exactly two spaces. If you have more or fewer spaces, or if you’re using tabs, you’ll get an error on startup.
# A deployment ensures pod(s) are restarted on failure
apiVersion: apps/v1
kind: Deployment
metadata:
name: memsql
spec:
replicas: 1 # only create one pod (container)
selector:
matchLabels:
app: memsql
template:
# Here's the definition of the pod:
metadata:
# The service finds all pods with matching metadata
labels:
app: memsql
spec:
containers:
- name: memsql
resources:
# Cluster-in-a-box image is pulled from Docker Hub
image: memsql/cluster-in-a-box
ports:
- containerPort: 3306 # MemSQL db
- containerPort: 8080 # MemSQL Studio
env:
# 'Y' means keep running after cluster init
- name: START_AFTER_INIT
value: 'Y'
# TODO: set to your desired password
- name: ROOT_PASSWORD
value: 'password'
# TODO: paste your license key from portal.memsql.com here:
- name: LICENSE_KEY
value: PASTE_YOUR_LICENSE_KEY_HERE
---
# A service load-balances across and routes traffic into pods
apiVersion: v1
kind: Service
metadata:
name: memsql
labels:
app: memsql
spec:
type: NodePort
# Find all pods
selector:
app: memsql
ports:
# MemSQL db port:
- name: '3306'
nodePort: 30306
port: 3306
targetPort: 3306
# MemSQL Studio port:
- name: '8080'
nodePort: 30080
port: 8080
targetPort: 8080
Open a new terminal window in the same directory with the kubernetes-memsql.yaml file. This could be Powershell, a command prompt, or a regular terminal.
Type this in the shell:
kubectl apply -f kubernetes-memsql.yamlThis tells Kubernetes to create (or adjust) the service and deployment definitions, and to startup the container. The output from the container isn’t streamed to the console.
To see the status of the pod as it starts up, type:
kubectl get all
