Merge branch 'develop' of https://github.com/HicServices/RDMP into de…

…velop

CHANGELOG.md

@@ -12,13 +12,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+
+- Upgraded Sql Server library from `System.Data.SqlClient` to `Microsoft.Data.SqlClient`
 - `ExecuteCommandAlterColumnType` now automatically alters \_Archive table too without asking for confirmation
 - When foreign key values are missing from lookups, the 'Missing' status is now attributed to the `_Desc` field (previously to the foreign key field)
 - Changed Console gui DLE / DQE (etc) execution to use ListView instead of TextView
 - Referencing an object by name in a script file now returns the latest when there are collisions e.g. "[ExtractableCohort]" would return the latest one (created during the script execution session)
 - Bump YamlDotNet from 11.2.0 to 11.2.1
 - Bump SecurityCodeScan.VS2019 from 5.1.0 to 5.2.1
 - Command 'Set' now shows as Impossible for property 'ID'
+- RDMP no longer complains about mixed capitalisation in server names and will connect using the capitalisation of the first encountered.
 
 ## Fixed
 
@@ -29,6 +32,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fixed bug where searching in console gui could be slow or miss keystrokes
 - Fixed bug in console gui where GoTo Project or Cohort would not highlight the correct item
 - Fixed bug in console gui where delete key was not handled resulting in a loop if errors occurred trying to delete the object
+- Removed limit of 500 characters on extraction SQL of columns
 
 ### Added
 
@@ -38,6 +42,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Pressing 'delete' key in console gui edit window now offers to set value of property to null
 - Editing a foreign key property (e.g. `PivotCategory_ExtractionInformation_ID`) now shows objects rather than asking for an `int` value directly
 - Fatal errrors in console gui now get logged by NLog (e.g. to console/file)
+- Added user setting `CreateDatabaseTimeout`
 
 ### Removed
 

Documentation/CodeTutorials/Packages.md

@@ -21,7 +21,7 @@
 | NPOI | [GitHub](https://github.com/tonyqus/npoi) | [2.4.1](https://www.nuget.org/packages/NPOI/2.4.1) | Apache 2.0 | Enables reading/writing Microsoft Excel files |
 | ExcelNumberFormat | [GitHub](https://github.com/andersnm/ExcelNumberFormat) | [1.1.0](https://www.nuget.org/packages/ExcelNumberFormat/1.1.0) |[MIT](https://opensource.org/licenses/MIT)  | Handles translating number formats from Excel formats into usable values | |
 | [NLog](https://nlog-project.org/) | [GitHub](https://github.com/NLog/NLog) | [4.7.10](https://www.nuget.org/packages/NLog/4.7.10) | [BSD 3-Clause](https://github.com/NLog/NLog/blob/dev/LICENSE.txt) | Flexible user configurable logging | |
-| HIC.FAnsiSql |[GitHub](https://github.com/HicServices/FAnsiSql) | [1.0.7](https://www.nuget.org/packages/HIC.FansiSql/1.0.7) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | [DBMS] abstraction layer |
+| HIC.FAnsiSql |[GitHub](https://github.com/HicServices/FAnsiSql) | [2.0.1](https://www.nuget.org/packages/HIC.FansiSql/2.0.1) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | [DBMS] abstraction layer |
 | HIC.BadMedicine | [GitHub](https://github.com/HicServices/BadMedicine) | [0.1.6](https://www.nuget.org/packages/HIC.BadMedicine/0.1.6) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | Generate Test Datasets for tests/exericses |
 | SSH.NET  | [GitHub](https://github.com/sshnet/SSH.NET)  | [2020.0.1](https://www.nuget.org/packages/SSH.NET/2020.0.1) | [MIT](https://github.com/sshnet/SSH.NET/blob/develop/LICENSE) | Enables fetching files from SFTP servers |
 | Moq 4 | [GitHub](https://github.com/moq/moq4) | [4.16.1](https://www.nuget.org/packages/Moq/4.16.1) |[BSD 3](https://github.com/moq/moq4/blob/master/License.txt)  | Mock objects during unit testing |
@@ -32,7 +32,7 @@
 | YamlDotNet | [GitHub](https://github.com/aaubry/YamlDotNet)  | [11.2.1](https://www.nuget.org/packages/YamlDotNet/11.2.1) | [MIT](https://opensource.org/licenses/MIT) |Loading configuration files|
 | [SecurityCodeScan.VS2019](https://security-code-scan.github.io/) | [GitHub](https://github.com/security-code-scan/security-code-scan) | [5.2.1](https://www.nuget.org/packages/SecurityCodeScan.VS2019/5.2.1) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html)| Performs static build time analysis for vulnerabilities in the codebase (e.g. Sql injection)| |
 | System.Runtime.Loader | [GitHub](https://github.com/dotnet/corefx) | [4.3.0](https://www.nuget.org/packages/System.Runtime.Loader/4.3.0) |[MIT](https://opensource.org/licenses/MIT)  | Allows loading assemblies in dot net core| |
-| System.Data.SqlClient | [GitHub](https://github.com/dotnet/corefx) | [4.8.2](https://www.nuget.org/packages/System.Data.SqlClient/4.8.2) | [MIT](https://opensource.org/licenses/MIT) | Enables interaction with Microsoft Sql Server databases |  |
+| Microsoft.Data.SqlClient | [GitHub](https://github.com/dotnet/SqlClient) | [3.0.0](https://www.nuget.org/packages/Microsoft.Data.SqlClient/3.0.0) | [MIT](https://opensource.org/licenses/MIT) | Enables interaction with Microsoft Sql Server databases |  |
 | System.Drawing.Common | [GitHub](https://github.com/dotnet/corefx)  | [5.0.2](https://www.nuget.org/packages/System.Drawing.Common/5.0.2) |[MIT](https://opensource.org/licenses/MIT) | Enables working with Bitmap resources |  |
 | System.Diagnostics.Debug | [GitHub](https://github.com/dotnet/corefx)  | [4.3.0](https://www.nuget.org/packages/System.Diagnostics.Debug/4.3.0) |[MIT](https://opensource.org/licenses/MIT) | Interact with Processes / Debug / Console |  |
 | System.IO.FileSystem.Primitives | [GitHub](https://github.com/dotnet/corefx) | [4.3.0](https://www.nuget.org/packages/System.IO.FileSystem.Primitives/4.3.0) |[MIT](https://opensource.org/licenses/MIT) | Provides common enumerations and exceptions for path-based I/O libraries |  |

Plugins/Plugin.Test/Plugin.Test.nuspec

@@ -14,7 +14,7 @@
         <copyright>Copyright 2018-2019</copyright>
         <dependencies>
 			      <dependency id="HIC.BadMedicine" version="0.1.6" />
-            <dependency id="HIC.FAnsiSql" version="1.0.7" />
+          <dependency id="HIC.FAnsiSql" version="2.0.1" />
             <dependency id="CsvHelper" version="19.0.0" />
             <dependency id="Newtonsoft.Json" version="13.0.1"/>
             <dependency id="CommandLineParser" version="2.8.0" />
@@ -29,7 +29,7 @@
             <dependency id="Xam.Plugins.Settings" version="3.1.1" />
             <dependency id="System.Runtime.Loader" version="4.3.0" />
             <dependency id="System.Drawing.Common" version="5.0.2" />
-            <dependency id="System.Data.SqlClient" version="4.8.2" />
+            <dependency id="Microsoft.Data.SqlClient" version="3.0.0" />
             <dependency id="System.Diagnostics.Debug" version="4.3.0" />
             <dependency id="System.IO.FileSystem.Primitives" version="4.3.0" />
             <dependency id="System.IO.FileSystem" version="4.3.0" />

Plugins/Plugin.UI/Plugin.UI.nuspec

@@ -14,7 +14,7 @@
         <copyright>Copyright 2018-2019</copyright>
         <dependencies>
 			<dependency id="HIC.BadMedicine" version="0.1.6" />
-            <dependency id="HIC.FAnsiSql" version="1.0.7" />
+            <dependency id="HIC.FAnsiSql" version="2.0.1" />
             <dependency id="DockPanelSuite.ThemeVS2015" version="2.13.0" />
             <dependency id="CsvHelper" version="19.0.0" />
 			<dependency id="NHunspell" version="1.2.5554.16953" />
@@ -29,7 +29,7 @@
             <dependency id="System.Runtime.Loader" version="4.3.0" />
             <dependency id="System.Drawing.Common" version="5.0.2" />
             <dependency id="Xam.Plugins.Settings" version="3.1.1" />
-            <dependency id="System.Data.SqlClient" version="4.8.2" />
+            <dependency id="Microsoft.Data.SqlClient" version="3.0.0" />
             <dependency id="System.Diagnostics.Debug" version="4.3.0" />
             <dependency id="System.IO.FileSystem.Primitives" version="4.3.0" />
             <dependency id="System.IO.FileSystem" version="4.3.0" />

Plugins/Plugin/Plugin.nuspec

@@ -17,13 +17,13 @@
             <dependency id="Newtonsoft.Json" version="13.0.1"/>
             <dependency id="CommandLineParser" version="2.8.0" />
             <dependency id="SSH.NET" version="2020.0.1" />
-            <dependency id="HIC.FAnsiSql" version="1.0.7" />
+            <dependency id="HIC.FAnsiSql" version="2.0.1" />
             <dependency id="NLog" version="4.7.10" />
             <dependency id="HIC.BadMedicine" version="0.1.6" />
             <dependency id="ExcelNumberFormat" version="1.1.0" />
             <dependency id="NPOI" version="2.4.1" />
             <dependency id="System.Runtime.Loader" version="4.3.0" />
-            <dependency id="System.Data.SqlClient" version="4.8.2" />
+            <dependency id="Microsoft.Data.SqlClient" version="3.0.0" />
             <dependency id="System.Drawing.Common" version="5.0.2" />
             <dependency id="Xam.Plugins.Settings" version="3.1.1" />
             <dependency id="System.Diagnostics.Debug" version="4.3.0" />

Rdmp.Core.Tests/CohortCreation/AggregateFilterPublishingTests.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using NUnit.Framework;
 using Rdmp.Core.Curation.Data;

Rdmp.Core.Tests/CohortCreation/QueryTests/JoinableCohortConfigurationTests.cs

@@ -6,7 +6,7 @@
 
 using System;
 using System.Data;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using System.Text.RegularExpressions;
 using FAnsi.Discovery;

Rdmp.Core.Tests/Curation/Anonymisation/ANOTableTests.cs

@@ -7,7 +7,7 @@
 using System;
 using System.Collections.Generic;
 using System.Data;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Diagnostics;
 using System.Linq;
 using System.Text.RegularExpressions;

Rdmp.Core.Tests/Curation/ImportTests/ShareLoadMetadataTests.cs

@@ -4,7 +4,7 @@
 // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using FAnsi.Discovery;
 using Moq;

Rdmp.Core.Tests/Curation/Integration/CredentialsTests.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using NUnit.Framework;
 using Rdmp.Core.Curation.Data;

Rdmp.Core.Tests/Curation/Integration/DataAccess/TestDataAccess.cs

@@ -6,7 +6,7 @@
 
 using System;
 using System.Collections.Generic;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using System.Threading;
 using FAnsi;
@@ -18,6 +18,7 @@
 using Rdmp.Core.Repositories.Managers;
 using ReusableLibraryCode;
 using ReusableLibraryCode.DataAccess;
+using ReusableLibraryCode.Exceptions;
 using Tests.Common;
 
 namespace Rdmp.Core.Tests.Curation.Integration.DataAccess
@@ -70,6 +71,30 @@ public void TestDistinctCredentials_UsernameMismatch()
 
         }
 
+
+        [Test]
+        public void TestDistinctCredentials_ServerMixedCapitalization_Allowed()
+        {
+            List<TestAccessPoint> testPoints = new List<TestAccessPoint>();
+
+            testPoints.Add(new TestAccessPoint("frank", "bob", null,null));
+            testPoints.Add(new TestAccessPoint("FRANK", "bob", null, null));
+
+            var server = DataAccessPortal.GetInstance().ExpectDistinctServer(testPoints.ToArray(), DataAccessContext.InternalDataProcessing, true);
+            Assert.AreEqual("frank", server.Name);
+        }
+
+        [Test]
+        public void TestDistinctCredentials_DatabaseMixedCapitalization_NotAllowed()
+        {
+            List<TestAccessPoint> testPoints = new List<TestAccessPoint>();
+
+            testPoints.Add(new TestAccessPoint("frank", "bob", null, null));
+            testPoints.Add(new TestAccessPoint("frank", "BOB", null, null));
+
+            var ex = Assert.Throws<ExpectedIdenticalStringsException>(() => DataAccessPortal.GetInstance().ExpectDistinctServer(testPoints.ToArray(), DataAccessContext.InternalDataProcessing, true));
+            StringAssert.Contains("All data access points must be into the same database, access points 'frankbob' and 'frankBOB' are into different databases", ex.Message);
+        }
         #endregion
 
         #region Distinct Connection String (from Collection tests - Passing)

Rdmp.Core.Tests/Curation/Integration/GovernanceTests.cs

@@ -6,7 +6,7 @@
 
 using System;
 using System.Collections.Generic;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using NUnit.Framework;
 using Rdmp.Core.Curation.Data;

Rdmp.Core.Tests/Curation/Integration/HangingConnectionTest.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Threading;
 using FAnsi.Discovery;
 using NUnit.Framework;

Rdmp.Core.Tests/Curation/Integration/QueryBuildingTests/AggregateBuilderTests/MicrosoftAggregateBuilderTests.cs

@@ -4,7 +4,7 @@
 // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using NUnit.Framework;
 using Rdmp.Core.Curation.Data;
 using Rdmp.Core.QueryBuilding;

Rdmp.Core.Tests/DataExport/ImportFileTests.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.IO;
 using FAnsi.Discovery;
 using NUnit.Framework;

Rdmp.Core.Tests/DataExport/TestExtractableTables.cs

@@ -6,7 +6,7 @@
 
 using System;
 using System.Collections;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using System.Reflection;
 using NUnit.Framework;

Rdmp.Core.Tests/DataLoad/Engine/Integration/BackfillTests.cs

@@ -6,7 +6,7 @@
 
 using System;
 using System.Collections.Generic;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using NUnit.Framework;
 using Rdmp.Core.Curation;

Rdmp.Core.Tests/DataLoad/Engine/Integration/DataLoadProgressUpdateInfoTests.cs

@@ -6,7 +6,7 @@
 
 using System;
 using System.Collections.Generic;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using Moq;
 using NUnit.Framework;
 using Rdmp.Core.Curation.Data;

Rdmp.Core.Tests/DataLoad/Engine/Integration/DatabaseOperationTests.cs

@@ -6,7 +6,7 @@
 
 using System;
 using System.Collections.Generic;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using FAnsi;
 using FAnsi.Discovery;
 using MapsDirectlyToDatabaseTable;

Rdmp.Core.Tests/DataLoad/Engine/Integration/HousekeepingTests.cs

@@ -4,7 +4,7 @@
 // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using FAnsi.Discovery;
 using NUnit.Framework;
 using Rdmp.Core.DataLoad.Triggers;

Rdmp.Core.Tests/DataLoad/Engine/Integration/JobDateGenerationStrategyFactoryTestsIntegration.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.IO;
 using FAnsi.Discovery;
 using NUnit.Framework;

Rdmp.Core.Tests/DataLoad/Engine/Integration/KVPAttacherTest.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.IO;
 using System.Linq;
 using NUnit.Framework;

Rdmp.Core.Tests/DataLoad/Engine/Unit/CommandLineHelperTests.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.IO;
 using FAnsi.Discovery;
 using FAnsi.Implementation;

Rdmp.Core.Tests/Logging/FatalErrorLoggingTest.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.IO;
 using System.Security.Cryptography;
 using FAnsi.Discovery;

Rdmp.Core.Tests/Logging/LogManagerTest.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using System.Threading.Tasks;
 using FAnsi.Discovery;

Rdmp.Core.Tests/QueryCaching/CachedAggregateConfigurationResultsManagerTests.cs

@@ -6,7 +6,7 @@
 
 using System;
 using System.Data;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using FAnsi.Discovery;
 using NUnit.Framework;

Rdmp.Core.Tests/Repositories/CatalogueRepositoryTests.cs

@@ -6,7 +6,7 @@
 
 using System;
 using System.Collections.Generic;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Text;
 using FAnsi.Implementation;
 using NUnit.Framework;

Rdmp.Core/CommandLine/DatabaseCreation/CataloguePipelinesAndReferencesCreation.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using Rdmp.Core.CohortCommitting.Pipeline.Destinations;
 using Rdmp.Core.CohortCommitting.Pipeline.Sources;

Rdmp.Core/CommandLine/DatabaseCreation/PlatformDatabaseCreation.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using FAnsi.Discovery;
 using MapsDirectlyToDatabaseTable.Versioning;
 using Rdmp.Core.CommandExecution;

Rdmp.Core/CommandLine/DatabaseCreation/PlatformDatabaseCreationOptions.cs

@@ -5,7 +5,7 @@
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
 using System.Collections.Generic;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using CommandLine;
 using CommandLine.Text;
 

Rdmp.Core/CommandLine/Options/RDMPCommandLineOptions.cs

@@ -4,7 +4,7 @@
 // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
 // You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
 
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using CommandLine;
 using Rdmp.Core.Repositories;
 using Rdmp.Core.Startup;

Rdmp.Core/Curation/TableInfoSynchronizer.cs

@@ -6,7 +6,7 @@
 
 using System;
 using System.Collections.Generic;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using System.Linq;
 using FAnsi.Discovery;
 using MapsDirectlyToDatabaseTable;

Rdmp.Core/DataExport/DataExtraction/Pipeline/Destinations/ExtractionDestination.cs

@@ -22,7 +22,7 @@
 using Rdmp.Core.DataExport.DataExtraction.UserPicks;
 using ReusableLibraryCode.DataAccess;
 using Rdmp.Core.Curation;
-using System.Data.SqlClient;
+using Microsoft.Data.SqlClient;
 using FAnsi.Discovery;
 using ReusableLibraryCode.Checks;