4.0.5.0-leisure

Added:
 - Add freedesktop.org desktop file and icon set #1438 (@a123b)
 - Add warning in help for blockchain scan for importprivkey #1469 (@jamescowens)
 - Consolidateunspent rpc function #1472 (@jamescowens)
 - Scraper 2.0 improvements #1481, #1488, #1509, and #1514 (@jamescowens, @cyrossignol)
   - explorer mode operation
   - simplified explainmagnitude output
   - improved convergence reporting, including scraper information in the tooltip when fDebug3 is set
   - improved statistics and SB contract core caching based on a bClean flag in the cache global
   - new SB format and packing for bv11
   - new SB contract hashing (native) for bv11
   - changes to accomodate new beacon approach
   - Implement in memory versioning for team file ETags
 - Implement local dynamic team requirement removal and whitelist #1502 (@cyrossignol)

Changed:
 - Quiet logging for getmininginfo and scraper INFO logging level #1460 (@jamescowens)
 - Spelling corrections #1461, #1462 (@caraka)
 - Update crypto module #1453 (@denravonska)
 - Update .travis.yml for Bionic #1475 (@jamescowens)
 - Create CPID classes and clean up CPID code #1477 (@cyrossignol)
 - Refactor researcher context and CPID harvesting #1480 (@cyrossignol)
   - Remove boinckey export RPC method and import handler
 - Notify when wallet locked in advertisebeacon RPC method #1504 (@cyrossignol)
 - Notify when wallet locked in beaconstatus RPC method #1506 (@cyrossignol)
 - Change spacer minimum height hint #1511 (@jamescowens)

Removed:
 - Remove safe mode #1434 (@denravonska)
 - Remove bitcoin.moc in Makefile.qt.include #1444 (@RoboticMind)
 - Clean up legacy Proof-of-Work functions #1497 (@cyrossignol)

Fixed:
 - Constrain walletpassphrase to 10000000 seconds #1459 (@jamescowens)
 - Straighten out localization in the scraper. #1471 (@jamescowens)
 - Quick fix for rainbymagnitude #1473 (@jamescowens)
 - Correct negation error in scraper tooltip for vScrapersNotPublishing #1484 (@jamescowens)
 - Fix staked block rejection when active researcher #1485 (@cyrossignol)
 - Add back informational magnitude to generated blocks #1489 (@cyrossignol)
 - Add back in the in sync check in ScraperGetNeuralContract #1492 (@jamescowens)
 - Scraper correct team file processing. #1501 (@jamescowens)
 - Have importwallet file path default to datadir #1508 (@jamescowens)
 - Scraper add Beacon Map size check to ensure convergence #1515 (@jamescowens)

.travis.yml

@@ -21,12 +21,12 @@ env:
     - PYTHON_DEBUG=1
     - WINEDEBUG=fixme-all
   matrix:
-# ARM - disabled for now, because Xenial has fPIC error with depends that can't be solved. Need to upgrade to Bionic, which is not available.
-    #- HOST=arm-linux-gnueabihf PACKAGES="g++-arm-linux-gnueabihf zlib1g-dev" GCCFLAGS="-fPIC" DEP_OPTS="NO_QT=1" CHECK_DOC=1 GOAL="install" GRIDCOIN_CONFIG="--enable-glibc-back-compat --disable-tests RUN_TESTS=false"
+# ARM
+    - HOST=arm-linux-gnueabihf PACKAGES="g++-arm-linux-gnueabihf zlib1g-dev" GCCFLAGS="-fPIC" DEP_OPTS="NO_QT=1" CHECK_DOC=1 GOAL="install" GRIDCOIN_CONFIG="--enable-glibc-back-compat --disable-tests RUN_TESTS=false"
 # Win32
     - HOST=i686-w64-mingw32 DPKG_ADD_ARCH="i386" DEP_OPTS="NO_QT=1" PACKAGES="python3 nsis g++-mingw-w64-i686 wine1.6" RUN_TESTS=false GOAL="" GRIDCOIN_CONFIG="--enable-reduce-exports"
 # Qt5 & system libs
-    - HOST=x86_64-unknown-linux-gnu PACKAGES="libqt5gui5 libqt5core5a qtbase5-dev libqt5dbus5 qttools5-dev qttools5-dev-tools libssl-dev libevent-dev bsdmainutils libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-iostreams-dev libboost-program-options-dev libboost-test-dev libboost-thread-dev libdb5.3++-dev libminiupnpc-dev protobuf-compiler libqrencode-dev xvfb zlib1g-dev" NO_DEPENDS=1 NEED_XVFB=1 RUN_TESTS=true GOAL="install" GRIDCOIN_CONFIG=" --with-incompatible-bdb --enable-reduce-exports --with-gui=qt5" 
+    - HOST=x86_64-unknown-linux-gnu PACKAGES="libqt5gui5 libqt5core5a qtbase5-dev libqt5dbus5 qttools5-dev qttools5-dev-tools libqt5charts5-dev libssl-dev libevent-dev bsdmainutils libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-iostreams-dev libboost-program-options-dev libboost-test-dev libboost-thread-dev libdb5.3++-dev libminiupnpc-dev protobuf-compiler libqrencode-dev xvfb zlib1g-dev" NO_DEPENDS=1 NEED_XVFB=1 RUN_TESTS=true GOAL="install" GRIDCOIN_CONFIG=" --with-incompatible-bdb --enable-reduce-exports --with-gui=qt5"
 # 32-bit + dash
     - HOST=i686-pc-linux-gnu PACKAGES="g++-multilib" DEP_OPTS="NO_QT=1" RUN_TESTS=false GOAL="install" GRIDCOIN_CONFIG="--enable-glibc-back-compat --enable-reduce-exports LDFLAGS=-static-libstdc++" USE_SHELL="/bin/dash"
 # Win64
@@ -36,10 +36,10 @@ env:
 
 before_install:
     - export PATH=$(echo $PATH | tr ':' "\n" | sed '/\/opt\/python/d' | tr "\n" ":" | sed "s|::|:|g")
-    - sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
+    #- sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
     - sudo apt-get update -q
-    - sudo apt-get install gcc-7 -y
-    - sudo apt-get install gcc-5-arm-linux-gnueabihf -y
+    #- sudo apt-get install gcc-7 -y
+    #- sudo apt-get install gcc-5-arm-linux-gnueabihf -y
 install:
     - if [ -n "$DPKG_ADD_ARCH" ]; then sudo dpkg --add-architecture "$DPKG_ADD_ARCH" ; fi
     - if [ -n "$PACKAGES" ]; then travis_retry sudo apt-get update; fi

CHANGELOG.md

@@ -4,6 +4,51 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](http://semver.org/).
 
+## [4.0.5.0] 2019-08-20, leisure
+### Added
+ - Add freedesktop.org desktop file and icon set #1438 (@a123b)
+ - Add warning in help for blockchain scan for importprivkey #1469 (@jamescowens)
+ - Consolidateunspent rpc function #1472 (@jamescowens)
+ - Scraper 2.0 improvements #1481, #1488, #1509, and #1514 (@jamescowens, @cyrossignol)
+   - explorer mode operation
+   - simplified explainmagnitude output
+   - improved convergence reporting, including scraper information in the tooltip when fDebug3 is set
+   - improved statistics and SB contract core caching based on a bClean flag in the cache global
+   - new SB format and packing for bv11
+   - new SB contract hashing (native) for bv11
+   - changes to accomodate new beacon approach
+   - Implement in memory versioning for team file ETags 
+ - Implement local dynamic team requirement removal and whitelist #1502 (@cyrossignol)
+
+### Changed
+ - Quiet logging for getmininginfo and scraper INFO logging level #1460 (@jamescowens)
+ - Spelling corrections #1461, #1462 (@caraka)
+ - Update crypto module #1453 (@denravonska)
+ - Update .travis.yml for Bionic #1475 (@jamescowens)
+ - Create CPID classes and clean up CPID code #1477 (@cyrossignol)
+ - Refactor researcher context and CPID harvesting #1480 (@cyrossignol)
+   - Remove boinckey export RPC method and import handler
+ - Notify when wallet locked in advertisebeacon RPC method #1504 (@cyrossignol)
+ - Notify when wallet locked in beaconstatus RPC method #1506 (@cyrossignol)
+ - Change spacer minimum height hint #1511 (@jamescowens)
+
+### Removed
+ - Remove safe mode #1434 (@denravonska)
+ - Remove bitcoin.moc in Makefile.qt.include #1444 (@RoboticMind)
+ - Clean up legacy Proof-of-Work functions #1497 (@cyrossignol)
+
+### Fixed
+ - Constrain walletpassphrase to 10000000 seconds #1459 (@jamescowens)
+ - Straighten out localization in the scraper. #1471 (@jamescowens)
+ - Quick fix for rainbymagnitude #1473 (@jamescowens)
+ - Correct negation error in scraper tooltip for vScrapersNotPublishing #1484 (@jamescowens)
+ - Fix staked block rejection when active researcher #1485 (@cyrossignol)
+ - Add back informational magnitude to generated blocks #1489 (@cyrossignol)
+ - Add back in the in sync check in ScraperGetNeuralContract #1492 (@jamescowens)
+ - Scraper correct team file processing. #1501 (@jamescowens)
+ - Have importwallet file path default to datadir #1508 (@jamescowens)
+ - Scraper add Beacon Map size check to ensure convergence #1515 (@jamescowens)
+
 ## [4.0.4.0] 2019-05-16, leisure
 ### Fixed
  - Adds back the new user wizard inadvertently removed #1464 (@jamescowens).

configure.ac

@@ -2,7 +2,7 @@ dnl require autoconf 2.60 (AS_ECHO/AS_ECHO_N)
 AC_PREREQ([2.60])
 define(_CLIENT_VERSION_MAJOR, 4)
 define(_CLIENT_VERSION_MINOR, 0)
-define(_CLIENT_VERSION_REVISION, 4)
+define(_CLIENT_VERSION_REVISION, 5)
 define(_CLIENT_VERSION_BUILD, 0)
 define(_CLIENT_VERSION_IS_RELEASE, true)
 define(_COPYRIGHT_YEAR, 2019)
@@ -234,12 +234,20 @@ if test "x$CXXFLAGS_overridden" = "xno"; then
   AX_CHECK_COMPILE_FLAG([-Wimplicit-fallthrough],[CXXFLAGS="$CXXFLAGS -Wno-implicit-fallthrough"],,[[$CXXFLAG_WERROR]])
 fi
 
-  # Check for optional instruction set support. Enabling these does _not_ imply that all code will
-  # be compiled with them, rather that specific objects/libs may use them after checking for runtime
-  # compatibility.
-  AX_CHECK_COMPILE_FLAG([-msse4.2],[[SSE42_CXXFLAGS="-msse4.2"]],,[[$CXXFLAG_WERROR]])
+enable_hwcrc32=no
+enable_sse41=no
+enable_avx2=no
+enable_shani=no
+
+if test "x$use_asm" = "xyes"; then
+
+# Check for optional instruction set support. Enabling these does _not_ imply that all code will
+# be compiled with them, rather that specific objects/libs may use them after checking for runtime
+# compatibility.
+AX_CHECK_COMPILE_FLAG([-msse4.2],[[SSE42_CXXFLAGS="-msse4.2"]],,[[$CXXFLAG_WERROR]])
 AX_CHECK_COMPILE_FLAG([-msse4.1],[[SSE41_CXXFLAGS="-msse4.1"]],,[[$CXXFLAG_WERROR]])
 AX_CHECK_COMPILE_FLAG([-mavx -mavx2],[[AVX2_CXXFLAGS="-mavx -mavx2"]],,[[$CXXFLAG_WERROR]])
+AX_CHECK_COMPILE_FLAG([-msse4 -msha],[[SHANI_CXXFLAGS="-msse4 -msha"]],,[[$CXXFLAG_WERROR]])
 
 TEMP_CXXFLAGS="$CXXFLAGS"
 CXXFLAGS="$CXXFLAGS $SSE42_CXXFLAGS"
@@ -301,6 +309,25 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
 )
 CXXFLAGS="$TEMP_CXXFLAGS"
 
+TEMP_CXXFLAGS="$CXXFLAGS"
+CXXFLAGS="$CXXFLAGS $SHANI_CXXFLAGS"
+AC_MSG_CHECKING(for SHA-NI intrinsics)
+AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
+    #include <stdint.h>
+    #include <immintrin.h>
+  ]],[[
+    __m128i i = _mm_set1_epi32(0);
+    __m128i j = _mm_set1_epi32(1);
+    __m128i k = _mm_set1_epi32(2);
+    return _mm_extract_epi32(_mm_sha256rnds2_epu32(i, i, k), 0);
+  ]])],
+ [ AC_MSG_RESULT(yes); enable_shani=yes; AC_DEFINE(ENABLE_SHANI, 1, [Define this symbol to build code that uses SHA-NI intrinsics]) ],
+ [ AC_MSG_RESULT(no)]
+)
+CXXFLAGS="$TEMP_CXXFLAGS"
+
+fi
+
 CPPFLAGS="$CPPFLAGS -DBOOST_SPIRIT_THREADSAFE -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS"
 
 AC_ARG_WITH([utils],
@@ -1115,6 +1142,7 @@ AM_CONDITIONAL([HARDEN],[test x$use_hardening = xyes])
 AM_CONDITIONAL([ENABLE_HWCRC32],[test x$enable_hwcrc32 = xyes])
 AM_CONDITIONAL([ENABLE_SSE41],[test x$enable_sse41 = xyes])
 AM_CONDITIONAL([ENABLE_AVX2],[test x$enable_avx2 = xyes])
+AM_CONDITIONAL([ENABLE_SHANI],[test x$enable_shani = xyes])
 AM_CONDITIONAL([USE_ASM],[test x$use_asm = xyes])
 
 AC_DEFINE(CLIENT_VERSION_MAJOR, _CLIENT_VERSION_MAJOR, [Major version])
@@ -1149,6 +1177,7 @@ AC_SUBST(PIE_FLAGS)
 AC_SUBST(SSE42_CXXFLAGS)
 AC_SUBST(SSE41_CXXFLAGS)
 AC_SUBST(AVX2_CXXFLAGS)
+AC_SUBST(SHANI_CXXFLAGS)
 AC_SUBST(LIBTOOL_APP_LDFLAGS)
 AC_SUBST(USE_UPNP)
 AC_SUBST(USE_QRCODE)

src/Makefile.am

@@ -34,6 +34,10 @@ if ENABLE_AVX2
 LIBGRIDCOIN_CRYPTO_AVX2 = crypto/libgridcoin_crypto_avx2.a
 LIBGRIDCOIN_CRYPTO += $(LIBGRIDCOIN_CRYPTO_AVX2)
 endif
+if ENABLE_SHANI
+LIBGRIDCOIN_CRYPTO_SHANI = crypto/libgridcoin_crypto_shani.a
+LIBGRIDCOIN_CRYPTO += $(LIBGRIDCOIN_CRYPTO_SHANI)
+endif
 
 # Make is not made aware of per-object dependencies to avoid limiting building parallelization
 # But to build the less dependent modules first, we manually select their order here:
@@ -70,8 +74,6 @@ GRIDCOIN_CORE_H = \
 	compat/endian.h \
 	contract/polls.h \
 	contract/contract.h \
-   contract/superblock.h \
-	cpid.h \
 	crypter.h \
 	db.h \
 	fwd.h \
@@ -86,10 +88,13 @@ GRIDCOIN_CORE_H = \
 	main.h \
 	miner.h \
 	mruset.h \
+	neuralnet/cpid.h \
 	neuralnet/neuralnet.h \
    neuralnet/neuralnet_native.h \
    neuralnet/neuralnet_stub.h \
 	neuralnet/project.h \
+	neuralnet/researcher.h \
+	neuralnet/superblock.h \
 	netbase.h \
 	net.h \
 	pbkdf2.h \
@@ -130,8 +135,6 @@ GRIDCOIN_CORE_CPP = addrman.cpp \
 	checkpoints.cpp \
 	contract/polls.cpp \
 	contract/contract.cpp \
-   contract/superblock.cpp \
-	cpid.cpp \
 	crypter.cpp \
 	db.cpp \
 	gridcoin.cpp \
@@ -144,7 +147,10 @@ GRIDCOIN_CORE_CPP = addrman.cpp \
    neuralnet/neuralnet.cpp \
    neuralnet/neuralnet_native.cpp \
    neuralnet/neuralnet_stub.cpp \
+	neuralnet/cpid.cpp \
 	neuralnet/project.cpp \
+	neuralnet/researcher.cpp \
+	neuralnet/superblock.cpp \
 	netbase.cpp \
 	net.cpp \
 	noui.cpp \
@@ -236,6 +242,12 @@ crypto_libgridcoin_crypto_avx2_a_CXXFLAGS += $(AVX2_CXXFLAGS)
 crypto_libgridcoin_crypto_avx2_a_CPPFLAGS += -DENABLE_AVX2
 crypto_libgridcoin_crypto_avx2_a_SOURCES = crypto/sha256_avx2.cpp
 
+crypto_libgridcoin_crypto_shani_a_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
+crypto_libgridcoin_crypto_shani_a_CPPFLAGS = $(AM_CPPFLAGS)
+crypto_libgridcoin_crypto_shani_a_CXXFLAGS += $(SHANI_CXXFLAGS)
+crypto_libgridcoin_crypto_shani_a_CPPFLAGS += -DENABLE_SHANI
+crypto_libgridcoin_crypto_shani_a_SOURCES = crypto/sha256_shani.cpp
+
 CTAES_DIST =  crypto/ctaes/bench.c
 CTAES_DIST += crypto/ctaes/ctaes.c
 CTAES_DIST += crypto/ctaes/ctaes.h

src/Makefile.qt.include

@@ -130,8 +130,7 @@ GRIDCOIN_MM = \
 
 QT_MOC = \
   qt/overviewpage.moc \
-  qt/rpcconsole.moc \
-  qt/bitcoin.moc
+  qt/rpcconsole.moc
 
 QT_QRC_CPP = qt/qrc_bitcoin.cpp
 QT_QRC = qt/bitcoin.qrc

src/Makefile.test.include

@@ -34,18 +34,20 @@ GRIDCOIN_TESTS =\
   test/beacon_tests.cpp \
   test/bignum_tests.cpp \
   test/block_tests.cpp \
-  test/cpid_tests.cpp \
-  test/crypter_tests.cpp \
   test/getarg_tests.cpp \
   test/gridcoin_tests.cpp \
   test/key_tests.cpp \
   test/mruset_tests.cpp \
   test/multisig_tests.cpp \
   test/netbase_tests.cpp \
+  test/neuralnet/cpid_tests.cpp \
   test/neuralnet/project_tests.cpp \
+  test/neuralnet/researcher_tests.cpp \
+  test/neuralnet/superblock_tests.cpp \
   test/rpc_tests.cpp \
   test/script_P2SH_tests.cpp \
   test/script_tests.cpp \
+  test/serialize_tests.cpp \
   test/sigopcount_tests.cpp \
   test/test_gridcoin.cpp \
   test/transaction_tests.cpp \

src/contract/contract.cpp

@@ -1,4 +1,3 @@
-#include "cpid.h"
 #include "init.h"
 #include "rpcclient.h"
 #include "rpcserver.h"

src/contract/polls.cpp

@@ -7,7 +7,6 @@
 #include "rpcclient.h"
 #include "rpcserver.h"
 #include "appcache.h"
-#include "cpid.h"
 #include "init.h" // for pwalletMain
 #include "block.h"
 
@@ -20,6 +19,8 @@ bool GetEarliestStakeTime(std::string grcaddress, std::string cpid);
 CBlockIndex* GetHistoricalMagnitude(std::string cpid);
 bool WalletOutOfSync();
 
+namespace NN { std::string GetPrimaryCpid(); }
+
 std::string GetShareType(double dShareType)
 {
     if (dShareType == 1) return "Magnitude";
@@ -83,10 +84,6 @@ std::pair<std::string, std::string> CreateVoteContract(std::string sTitle, std::
     //6-20-2015
     double nBalance = GetTotalBalance();
     uint256 hashRand = GetRandHash();
-    std::string email = GetArgument("email", "NA");
-    boost::to_lower(email);
-    GlobalCPUMiningCPID.email = email;
-    GlobalCPUMiningCPID.lastblockhash = GlobalCPUMiningCPID.cpidhash;
     if (!PollExists(sTitle))
         return std::make_pair("Error", "Poll does not exist.");
     if (PollExpired(sTitle))
@@ -126,7 +123,7 @@ std::pair<std::string, std::string> CreateVoteContract(std::string sTitle, std::
         return std::make_pair("Error", "Sorry, When voting in a Both Share Type poll, your stake age Or your CPID age must be older than the poll duration.");
     else
     {
-        std::string voter = "<CPIDV2>"+GlobalCPUMiningCPID.cpidv2 + "</CPIDV2><CPID>"
+        std::string voter = "<CPID>"
                 + GlobalCPUMiningCPID.cpid + "</CPID><GRCADDRESS>" + GRCAddress + "</GRCADDRESS><RND>"
                 + hashRand.GetHex() + "</RND><BALANCE>" + RoundToString(nBalance,2)
                 + "</BALANCE><MAGNITUDE>" + RoundToString(dmag,0) + "</MAGNITUDE>";
@@ -291,23 +288,24 @@ std::string PollAnswers(std::string pollname)
 }
 std::string GetProvableVotingWeightXML()
 {
+    std::string primary_cpid = NN::GetPrimaryCpid();
     std::string sXML = "<PROVABLEMAGNITUDE>";
     //Retrieve the historical magnitude
-    if (IsResearcher(msPrimaryCPID))
+    if (IsResearcher(primary_cpid))
     {
-        StructCPID& st1 = GetLifetimeCPID(msPrimaryCPID);
-        CBlockIndex* pHistorical = GetHistoricalMagnitude(msPrimaryCPID);
+        StructCPID& st1 = GetLifetimeCPID(primary_cpid);
+        CBlockIndex* pHistorical = GetHistoricalMagnitude(primary_cpid);
         if (pHistorical->nHeight > 1 && pHistorical->nMagnitude > 0)
         {
             std::string sBlockhash = pHistorical->GetBlockHash().GetHex();
             std::string sError;
             std::string sSignature;
-            bool bResult = SignBlockWithCPID(msPrimaryCPID, pHistorical->GetBlockHash().GetHex(), sSignature, sError);
+            bool bResult = SignBlockWithCPID(primary_cpid, pHistorical->GetBlockHash().GetHex(), sSignature, sError);
             // Just because below comment it'll keep in line with that
             if (!bResult)
                 sSignature = sError;
             // Find the Magnitude from the last staked block, within the last 6 months, and ensure researcher has a valid current beacon (if the beacon is expired, the signature contain an error message)
-            sXML += "<CPID>" + msPrimaryCPID + "</CPID><INNERMAGNITUDE>"
+            sXML += "<CPID>" + primary_cpid + "</CPID><INNERMAGNITUDE>"
                     + RoundToString(pHistorical->nMagnitude,2) + "</INNERMAGNITUDE>" +
                     "<HEIGHT>" + ToString(pHistorical->nHeight)
                     + "</HEIGHT><BLOCKHASH>" + sBlockhash + "</BLOCKHASH><SIGNATURE>" + sSignature + "</SIGNATURE>";