Modern Dev Infrastructure #1373
Comments
|
Since OOS and Workfklow Manager are only needed for very specific resources, we don't need these machines that badly from the beginning, right? |
|
If we need to allow users to access SharePoint sites from a browser running on the same SharePoint machine, we need also DisableLoopbackCheck, right? |
|
It might be reasonable to use just one machine for both |
|
I love this idea! Would be great if we offer a method to quickly provision predefined configurations!
Agree, that would be future a nice to have
Absolutely! Either that or the BackConnectionHostNames, which allows for a more granular configration. DisableLoopbackCheck basically disables the entire check. More info here: https://docs.microsoft.com/en-us/troubleshoot/windows-server/networking/accessing-server-locally-with-fqdn-cname-alias-denied
Shouldn't we also add the possibility to provision in an existing AD? |
|
Yes, provisioning in an existing might be needed. Do you think it will be often case from the contributor role perspective? As a contributor, do I often need to provision new dev environment in an existing AD? Won't it be tricky to provision Azure VMs in existing domain? Will this require administrative privileges in the domain? |
|
One of the developers of https://github.com/automatedlab/automatedlab here 👋 the inventor being @raandree Many items on the list @shurick81 provided can be accomplished using AutomatedLab. At the moment targeting Hyper-V or Azure. If Azure is used, the host system can be Linux and Windows, for Hyper-V it is obviously Windows. Internet-connectivity is strongly recommended for the host, since AutomatedLab often downloads additional files during deployments. Additionally, ISO files are necessary for the OSses and for most products, including SharePoint. A dev and CI-environment could look like this: #requires -runAs
# Automation: Configure telemetry opt-in or opt-out
[Environment]::SetEnvironmentVariable('AUTOMATEDLAB_TELEMETRY_OPTIN', 'yes', 'Machine')
Install-Module AutomatedLab -Force -AllowClobber
Enable-LabHostRemoting -Force
New-LabSourcesFolder -Drive C -Force # Or wherever you want to store the ISO files!
# Copy all required ISO files to
"$labsources\ISOs" # $labSources is a dynamic variable and will point to lab sources
$labName = 'SPDev'
$domainName = 'contoso.com'
New-LabDefinition -Name $labname -DefaultVirtualizationEngine Azure
Add-LabDomainDefinition -Name $domainName -AdminUser Install -AdminPassword Somepass1
Set-LabInstallationCredential -Username Install -Password Somepass1
$PSDefaultParameterValues = @{
'Add-LabMachineDefinition:ToolsPath'= "$labSources\Tools"
'Add-LabMachineDefinition:DomainName' = 'contoso.com'
'Add-LabMachineDefinition:OperatingSystem' = 'Windows Server 2019 Datacenter'
}
# Domain
$postInstallActivity = @()
$postInstallActivity += Get-LabPostInstallationActivity -ScriptFileName 'New-ADLabAccounts 2.0.ps1' -DependencyFolder $labSources\PostInstallationActivities\PrepareFirstChildDomain
$postInstallActivity += Get-LabPostInstallationActivity -ScriptFileName PrepareRootDomain.ps1 -DependencyFolder $labSources\PostInstallationActivities\PrepareRootDomain
Add-LabMachineDefinition -Name SPDC1 -Memory 1GB -Roles RootDC -PostInstallationActivity $postInstallActivity
# CA
Add-LabMachineDefinition -Name SPCA1 -Memory 1GB -Roles CARoot
# WAC Server, Tools machine
Add-LabMachineDefinition -Name SPWAC1 -Memory 1GB -Roles WindowsAdminCenter
# SQL Server
Add-LabIsoImageDefinition -Name SQLServer2016 -Path $labSources\ISOs\en_sql_server_2016_standard_with_service_pack_2_x64_dvd_12124191.iso
Add-LabMachineDefinition -Name SPDB1 -Roles SQLServer2016 -Memory 2GB
Install-Lab
Show-LabDeploymentSummaryAll customizations can be done after the deployment using Invoke-LabCommand, e.g.: $vsCodeDownloadUrl = 'https://go.microsoft.com/fwlink/?Linkid=852157'
$gitDownloadUrl = 'https://github.com/git-for-windows/git/releases/download/v2.34.1.windows.1/Git-2.34.1-64-bit.exe'
$vscodePowerShellExtensionDownloadUrl = 'https://marketplace.visualstudio.com/_apis/public/gallery/publishers/ms-vscode/vsextensions/PowerShell/2021.10.2/vspackage'
$chromeDownloadUrl = 'https://dl.google.com/tag/s/appguid%3D%7B8A69D345-D564-463C-AFF1-A69D9E530F96%7D%26iid%3D%7BC9D94BD4-6037-E88E-2D5A-F6B7D7F8F4CF%7D%26lang%3Den%26browser%3D5%26usagestats%3D0%26appname%3DGoogle%2520Chrome%26needsadmin%3Dprefers%26ap%3Dx64-stable-statsdef_1%26installdataindex%3Dempty/chrome/install/ChromeStandaloneSetup64.exe'
$notepadPlusPlusDownloadUrl = 'https://github.com/notepad-plus-plus/notepad-plus-plus/releases/download/v8.1.9.3/npp.8.1.9.3.Installer.exe'
$vscodeInstaller = Get-LabInternetFile -Uri $vscodeDownloadUrl -Path $labSources\SoftwarePackages -PassThru
$gitInstaller = Get-LabInternetFile -Uri $gitDownloadUrl -Path $labSources\SoftwarePackages -PassThru
Get-LabInternetFile -Uri $vscodePowerShellExtensionDownloadUrl -Path $labSources\SoftwarePackages\VSCodeExtensions\ps.vsix
$chromeInstaller = Get-LabInternetFile -Uri $chromeDownloadUrl -Path $labSources\SoftwarePackages -PassThru
$notepadPlusPlusInstaller = Get-LabInternetFile -Uri $notepadPlusPlusDownloadUrl -Path $labSources\SoftwarePackages -PassThru
Install-LabSoftwarePackage -Path $notepadPlusPlusInstaller.FullName -CommandLine /S -ComputerName SPWAC1
Install-LabSoftwarePackage -Path $vscodeInstaller.FullName -CommandLine /SILENT -ComputerName SPWAC1
Install-LabSoftwarePackage -Path $gitInstaller.FullName -CommandLine /SILENT -ComputerName SPWAC1
Install-LabSoftwarePackage -Path $chromeInstaller.FullName -ComputerName SPWAC1 -CommandLine '/silent /install'
Copy-LabFileItem -Path $labSources\SoftwarePackages\VSCodeExtensions -ComputerName SPWAC1Your scenarios sound like one script and a ValdidateSet-Parameter specifying the scenario and then using Add-LabMachine a bunch of times. As an alternative, the |
|
Do we need these machines available from the Internet via HTTP(S) or all web requests will be probably done internally? If it's only internal requests, then I think it will be easier with DNS and certificates: hosts files and self-signed certificates. |
|
@shurick81 Initially I would say internal only. If there are needs later on, we can always expand this to external access as well later on. @nyanhp Thanks for this info, it is a great start to see how we can use AutomatedLab to deploy SharePoint environments! |
|
@nyanhp if we want to start with Azure, do you know of any kind of comparison of AutomatedLab and similar provisioning tools like Vagrant/Terraform/Bicep/ARM templates etc.? |
|
Just to add something to this discussion: Lately I started to use this ARM template to generate test-environments and I was happy with the outcome: https://github.com/Azure/azure-quickstart-templates/tree/master/application-workloads/sharepoint/sharepoint-adfs Maybe I would be beneficial to include @Yvand in this discussion. |
|
From an operations perspective, it would be great to include these recommendations about service accounts into the template: https://docs.microsoft.com/en-us/sharepoint/install/account-permissions-and-security-settings-in-sharepoint-server-2016#service-account-recommendations Over the years I have seen way to many SharePoint DEV environments that were using a single account. Obviously everything worked perfectly :-) |
|
@shurick81 Bicep/ARM templates are template languages as far as I am concerned, and are a way to use Infrastructure as Code on Azure. The template functions that exist are not what I would consider tools. AutomatedLab uses an ARM template if you deploy to Azure, which can be exported if necessary. Bicep is an abstraction for ARM. Neither Bicep nor ARM can really help with the customization apart from onboarding to Azure Automation DSC or applying Azure Guest Configuration. I have no experience with Vagrant. Terraform is nice for deployments, but good luck with the customizations afterwards. AutomatedLab not only deploys VM workloads (which would be the same as ARM/Bicep/terraform/Vagrant/InsertTechnologyHere), but also does the installation and configuration of a broad range of Microsoft products including SharePoint, and provides custom cmdlets to interact with the VMs afterwards, no credential handling required. If you don't want to use AutomatedLab, but e.g. an ARM template, I would recommend building the configuration in your pipeline and publishing to an Azure Automation Account, including all resource modules required. In the ARM template, you would onboard the node to use Azure Automation DSC. The same would be true with regards to terraform - you need to find some way of applying the MOF. |
|
@nyanhp thanks a lot for the detailed overview! Yes, this was pretty much my impression as well. I'm used to provision using Terraform and Vagrant, but they both lack orchestration. For example, you can define 8 VMs to be created, the order/parralelism they should be created with, but then you can only set up the sequence of the scripts running on these VMs. For example, for provisioning you often need something like this:
And if you want to have this in Terraform or ARM, you will need to develop your own provisioning with such orchestration. For example, as you say, in such case we could run a pipeline, for example with Azure Automation Account or even regular Azure DevOps pipeline. An alternative might be executing on one of the VMs some PS script that will do such orchestration. I think we should consider AutomatedLab as a primary option here, especially taking into account that this ticket is just one of the phases:
|
|
One more idea to the requirements list might be disabling defender or applying the antivirus policies that are required for SharePoint: https://support.microsoft.com/en-us/office/certain-folders-may-have-to-be-excluded-from-antivirus-scanning-when-you-use-file-level-antivirus-software-in-sharepoint-01cbc532-a24e-4bba-8d67-0b1ed733a3d9 |
|
I like the idea to add the exclusions. We can use this script as a starting point: https://github.com/ykuijs/Powershell/blob/dev/SharePoint/SP_AV_Exclusions.ps1 I rather not add a feature to disable a security feature like Defender......if that is what the user wants he/she should do it manually. |
|
@nyanhp thanks a lot for your suggestions and guidance and of course for contributing to AL, now I deployed a few machines with AL and so far there seems no any showstoppers for us to use it for SharePointDsc! Here's the code I used: Connect-AzAccount
Set-AzContext -Tenant 85173d93-99ef-4dff-9b45-495719659133
New-LabDefinition -Name 'SharePointDscDev00' -DefaultVirtualizationEngine Azure
Add-LabAzureSubscription -SubscriptionName RND-DEV-SSG_SWEDEN_EVAL_MS_SHP_SE -DefaultLocationName "west europe"
# Answer Y
Add-LabDomainDefinition -Name 'contoso.com' -AdminUser Install -AdminPassword Somepass1
Set-LabInstallationCredential -Username Install -Password Somepass1
$PSDefaultParameterValues = @{
'Add-LabMachineDefinition:ToolsPath'= "$labSources\Tools"
'Add-LabMachineDefinition:DomainName' = 'contoso.com'
'Add-LabMachineDefinition:OperatingSystem' = 'Windows Server 2019 Datacenter (Desktop Experience)'
}
# Domain
$postInstallActivity = @()
$postInstallActivity += Get-LabPostInstallationActivity -ScriptFileName 'New-ADLabAccounts 2.0.ps1' -DependencyFolder $labSources\PostInstallationActivities\PrepareFirstChildDomain
$postInstallActivity += Get-LabPostInstallationActivity -ScriptFileName PrepareRootDomain.ps1 -DependencyFolder $labSources\PostInstallationActivities\PrepareRootDomain
Add-LabMachineDefinition -Name swazspdc00 -Memory 1GB -Roles RootDC -PostInstallationActivity $postInstallActivity -AzureProperties @{RoleSize = 'Standard_B2s'}
Add-LabMachineDefinition -Name swazspwac00 -Memory 1GB -Roles WindowsAdminCenter -AzureProperties @{RoleSize = 'Standard_D2s_v3'}
Add-LabIsoImageDefinition -Name SQLServer2016 -Path $labSources\ISOs\SQLServer2016-x64-ENU.iso
Add-LabMachineDefinition -Name swazspdb00 -Roles SQLServer2016 -Memory 2GB -AzureProperties @{RoleSize = 'Standard_D2s_v3'}
Add-LabMachineDefinition -Name swazspsvr00 -Memory 8GB -AzureProperties @{RoleSize = 'Standard_D2s_v3'}
Install-Lab
&(Get-LabVMRdpFile -ComputerName swazspwac00)A question though, why we need a machine with WindowsAdminCenter role? |
|
By the way, I also looked into Azure Lab Services but this functionality mostly focuses on scheduling and usage quotes for VMs. I could not notice any features regarding provisioning software on the lab VMs. |
|
@nyanhp, could you suggest the roles to use for the set of 4 machines in the lab:
SQL instance with necessary parallelism configured (MAXDOP) |
|
@ykuijs from your experience of contributing to this project, what do you think would be the most important topology. The topology that you would benefit the most from:
|
|
Option 3
From: Aleksandr SaPozhkov ***@***.***>
Sent: Friday, 4 February 2022 08:57
To: dsccommunity/SharePointDsc ***@***.***>
Cc: Subscribed ***@***.***>
Subject: Re: [dsccommunity/SharePointDsc] Modern Dev Infrastructure (Issue #1373)
@ykuijs<https://github.com/ykuijs> from your experience of contributing to this project, what do you think would be the most important topology. The topology that you would benefit the most from:
1. One machine
* AD, SQL, dev tools and empty space for installing SP - everything on one machine
1. Separate AD machine
* AD
* SQL, dev tools and empty space for installing SP
1. Separate SQL machine
* AD
* SQL
* dev tools and empty space for installing SP
1. Double SP
* AD
* SQL
* empty machine for SP
* empty machine for SP
* dev tools
—
Reply to this email directly, view it on GitHub<#1373 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AID6F4B4Y7ORX6EZ2ZQDONDUZOBGLANCNFSM5MBPHOCQ>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>
|
|
What do you think about the following parametrization for starters:
This seems like the minimum input from user that is required for provisioning first installations I guess in first versions of this new environment we can skip the following parametrization (hardcode it for now):
|
|
Hi everyone, I follow this discussion with some curiosity since @andikrueger added me, and also because I didn't know about AutomatedLab. |
|
I have been testing with AutomatedLab to deploy a new environment in Azure. What I am deploying at the moment is:
I have also been looking at configuring SQL with SharePoint specific settings like MaxDOP. For this I use SQLServerDsc, which works great. We can use DSC to fully configure SharePoint as well. I have created a another repo to share the work I have been doing so far, which you can find here. Am currently working on some more tests, after which I will push my changes to this repo later today. @Yvand, the idea is to create a simple solution to quickly deploy SharePoint environments with a certain configuration, both to Azure and Hyper-V (and potentially other platforms in the future). Of course we can use ARM templates for Azure, but for Hyper-V this will be a little more difficult. Second: Personally I find creating ARM templates very complex, so if I want to deploy a different type of topology, I get lost in creating the ARM template. And as far as I know, ARM templates are limited in the post processing options. That is where AutomatedLab comes in. It is a generic solution that works both on Azure and Hyper-V and at the same time has a lot of post processing possibilities to fully configure an environment. That is why we are looking at using that option. |
|
When it comes to the levels of SharePoint provisioning, I think we need a few options also:
Or maybe 3 is not needed and we can just use option 4 for the same cases as I described for option 3? Logically it seems like a good idea to start with option 1 as the easiest and then we can put it already to the usage and we could even set up a simple SharePointDsc CI that will only test using |
ykuijs
added
enhancement
|
@ykuijs when you want to commit a fix or improvement in SharePointDsc, what version(s) of SharePoint do you generally use for debugging/testing? It might be sensible to use at least two opposite versions, like the oldest supported, SharePoint 2013 and the newest, SharePoint Subscription Edition? |
|
Ideally I test changes in all supported SharePoint version, but this just isn't always possible. I have various test environments available in Azure, but I usually test on the most recent version of SharePoint and on one older version. I am thinking about setting up an integration test environment in which I can easily run a full test against all SharePoint versions. Our Modern Dev Infrastructure idea could be very suitable for quickly creating that as well. |
|
yes, setting up test environments for SharePointDsc is one of the use cases we are working on here in this thread! |
|
I have now prepared the infra for developing/debugging/testing the following resources: SPInstallPrereqs, SPInstall, SPInstallLanguagePack, SPProductUpdate and SPDocIcon. Let's give it a try @ykuijs in something close to real life scenario and for setting up some CI for SharePointDsc? |
Since https://github.com/dsccommunity/SharePointDsc/wiki/Creating-an-Azure-development-environment is a bit outdated and was not improved for some time, I thought we could rework the infra. Here are ideas for the requirements.
As a SharePointDsc contributor, I want to have a clear routine for creating different environments that I could then use for debugging SharePointDsc commits.
For the beginning, the routine should allow to create environments in Azure. Later on it should also allow using Hyper-V, AWS, GCP, VirtualBox, VMWare Player, etc.
The routine should allow to create at least the following topologies:
One server with AD, SQL and dev tools
2 servers:
AD
SQL and dev tools
3 servers:
AD
SQL
dev tools
4 servers:
AD
SQL
1 empty server
dev tools
5 servers:
AD
SQL
OOS farm
1 empty server
dev tools
8 servers:
AD
SQL
OOS farm
3 Workflow Manager servers
1 empty server
dev tools
It should be easy to add more empty servers.
The code should allow provisioning the following Windows Server versions:
Windows Server 2012 R2
Windows Server 2016
Windows Server 2019
Windows Server 2022
Windows Server 2019 Core
Windows Server 2022 Core
The code should allow provisioning the following SQL versions:
SQL Server 2014 with May 2014 CU or later
SQL Server 2016
SQL Server 2017
SQL Server 2019
Dev tools:
VisualStudio Code
Git
SSMS
ULSViewer
AD Remote Administration
The code should implement additional SharePoint requirements:
Known domain admin account
SP install account in AD
SP Service accounts in AD
An AD group that represents SP administrator group. Install account is a member of the group.
A few OUs for testing AD sync
Sync account should have Replicating Directory Changes
On each member of domain, the SP administrators group should be included in the machine local administrators group.
SP administrator group or install account should have sysadmin role on the SQL instance
SQL Maximum Degree of Parallelism should be set according to SP requirements
SSL certificates
For the beginning it should be possible to run the code from Windows. Later on it should be possible to execute from MacOS and Linux laptops.
The text was updated successfully, but these errors were encountered: