Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
dcrpg (DB v2) - PostgreSQL backend and Lite mode (#209)
* Start packages dcrpg and dbtypes, and app cmd/rebuilddb2.
move dcrsqlite to db subdir
Update dcrsqlite imports.
Create package dbtypes with more thorough data types than dcrdatapi types.
Create JSONB type with driver.Valuer and sql.Scanner implemeted for PostgreSQL support.
Define tables in dcrpg.
Create dcrpg.Connect() and dcrpg.CreateTables().
Start cmd/rebuilddb2.
Add --droptables flag, existence check, version in table comments.
Improve CreateTables().
block/tx/vin/vout struct extraction demo in rebuilddb2
keep converging on db table defs
Add pg inserts for vout and tx, store PKs in parents.
Add queries.go, statements.go and stmtinternal.go in dcrpg.
dcrpg: Move table statements test into internal.
Move Connect into own file.
Move some make*InsertStatement functions into internal.
dcrpg: full scan. block_chain table for tracking prev/next block hashes.
rebuilddb2 does a full chain scan. But set synchronous_commit to off in postgresql.conf!!!!
Add InsertVouts to add multiple vouts in one db tx.
Add InsertBlockPrevNext and UpdateBlockNext for the block_chain table.
Move processTransactions to extraction.go.
Periodically log tx and vout processing rate.
Make -D a short arg to --droptables.
Fix millisecond logging.
Add PrevOut to VinTxProperty, UPSERT stuff, and update logging.
Start drafting RetrieveSpendingTxs and RetrieveSpendingTx, to get a tx given and outpoint and funding tx, respectively.
Reorganize scan in rebuilddb2.
Add unique_hashes uniqueness constraint for transactions table.
rebuilddb2/dbtypes/dcrpg: Working retrieve-spending, indexing, vin table, etc.
Add vin table and InsertVin[s] functions in dcrpg.
Fix (*VinTxPropertyARRAY).Scan that had to work with []interface{} and map[string]interace{} right after json.Unmarshal.
Add Index/Deindex functions in dcrpg.
Add RetrieveSpending/Funding functions and pg statements in dcrpg.
Make pg inset statements that optionally ckeck for conflicts instead of error, and optionally upsert.
Add Tree to VinTxProperty, and tree to pg transactions table.
Add tx_hash and tx_index to vin table (and VinTxProperty).
Add IsStakeTx to txhelpers.
support UNIX domain socket connection to pg
Change vout structure, indexes for vout table.
dcrpg: vout_t insert into transactions table.
Insert vout array usign makeARRAYOfVouts.
create and check existence of type statements, called in rebuilddb2.
transaction in blocks selection queries.
vout value retrieval queries.
Configure speedReport function to run after db inserts or on bail out using sync.Once.
Be sure startHeight is 0 on empty tables, not 1.
Add ResumeInitSync config option for disabling conflict checks regardless of db height.
Also play with concurrent stake and regular transaction/vin/vout db insertion.
Fix Vout values queries, make ResumeInitSync drop/create indexes.
Old (wrong) RetrieveVoutValues is mystery, but the new one is good.
voutindex+1 is the index in sql.
Remove ON CONFLICT check from insertVoutRowChecked sicne there does not seem to be a useful UNIQUE INDEX.
typos
Add UNIX domain socket example to sample-rebuilddb2.conf
Add README.md for dcrpg.
Add dbtypes.MsgBlockToDBBlock into new conversion.go to reuse this code.
Bump version to 0.9.0.
fix password psql login
* ChainDB wrapping sql.DB, dcrpg package.
Collect/Store MsgBlock too, make ChainDB to wrap the sql.DB. INCOMPLETE!
Add several functions to ChainDB for getting data.
TODO: Cache DB PKs in ChainDB somewhere/somehow when we get them while getting the actual data.
Use the new ChainDB functions in rebuilddb2.
Also store Vouts in dbtypes.Tx.Vouts inside storeTxns instead of copying over later. TODO: remove dbTxVouts from ExtractBlockTransactions?
Remove extra newlines in logging.
Add dcrpg backend to dcrdata, change mainCore to return error, not int.
Add PostgreSQL tuning reference.
Move testing stuff from rebuilddb2 to a _test file.
Hack together a simultaneous sync of PG adn sqlite DBs.
Make async goroutines wrapping the regular sync functions, but sending the results (height and error) on a channel.
Add a height return to dcrsqlite's sync function.
Init PSQL logger, rename DSQL->SQLT.
More logging in DB resync loop in main.
Fix two incorrect returned heights from (*ChainDB).SyncChainDB.
Return tx index in block from RetrieveTxByHash, and tx indexes from RetrieveTxsByBlockHash.
Order in the returned tx slice is not in the order of vout index, so you need this information.
Build tag pgblockchain_test.go so it doesn't automatically run. It runs with sync'd mainnet db, so not an faast test from scratch.
* Remove lots of pointless RPC calls.
Add TicketTxnsInBlock.
Remove left over glide files
* dcrd deps update for db2
Update original rebuilddb for height output of SyncDBWithPoolValue.
* add postgresql config to sample dcrdata.conf
* Accelerate vin/vout dbtx, use PG array access proper, vout retrieval only from vout table not transactions.
Use prepared statements now that sprintf injection is no longer needed for array access, accelerating vin/vout insertion by ~1.5x.
Add batch Tx insertion, when combined with above gives total of ~2x speed up.
RetrieveVoutValue, etc. are now FROM vouts instead of trying to use the crazy JSONB or ROW types that were under testing. Update VoutValue and VoutValues accordingly.
Stop storing the actual vin/vout data in the transactions table, just the PKs for them in their own tables.
Use tx tree for transaction queries, include in index to make unique.
Fix dcrdata panic on non-existing cli flag.
UInt64Array in new arrays.go is based on Int64Array in lib/pq's array.go.
Fix tests in pgblockchain_test.go (run with "go test -tags mainnettest -v" after syncing with mainnet).
* Add spent link to Transactions page, calling SpendingTransaction.
Give dcrdata the start of a secondary data source.
Call SpendingTransaction from txPage to get the information about spending transactions for each outpoint.
Retrieve vin and vout tx inds when looking up spent tx.
Ignore bogus testnet2 organiation address.
* Add "Lite" mode to use only sqlite.
Rename PGHostPort -> PGHost to be less confusing when using UNIX sockets.
Update sample-dcrdata.conf with "lite" flag and renamed pghost flag.
self nil pointer check on ChainDB Store and SyncChainDBAsync methods to prevent panics in lite mode. In lite mode, SyncChainDBAsync returns -1 height and appropriate message.
explorer has liteMode bool field, set in constructor by checking if explorerSource is a nil interface or has a nil pointer value.
The first page with different output depending on lite mode is txPage, for the spending tx links.
* dcrpg: added address table, tree columns added to vins.
- Add AddressRow struct, with address, funding tx details, vout PK, value, spending tx details, and vin PK.
- Add create and drop index statements and functions.
- Add address statements (addrstmts.go) and query functions.
- Block processing now builds slice of AddressRow structs:
1. Use InsertVouts to create an AddressRow slice with just the spending tx details set.
2. Store it.
3. Query pg for funding details (prevouts) for each of the new Vins.
4. Lookup the address rows for these prevouts.
5. Set the spending tx details (txns and vins processed in current block).
- Add tree to output of SpendingTransaction.
- Add tx_tree and prev_tx_tree columns to vins table.
- Organize vinoutstmts.go content.
Add TxTree field to VinTxProperty, skip lookup of coinbase in addresses table.
Remove stray comma in CreateVinTable.
Flag to not update spending info in addresses table (no update by default).
Still hard coded to update when running normally.
Remove UNIQUE from some indexes that cannot be so, for now.
forgot funding tx vout index in addresses table.
Combine address and vins table queries for efficiency.
close statement and rollback on early return
* Add README.md for rebuilddb2
* Update README.md for db2 and v0.9.0
* Docs, cleanup, delint, compress/align some structs.
* Give http a few seconds to bind, waiting for an error.
Previously we just fired it off and didn't automatically quit dcrdata if the web server failed to bind.
* dcrpg - Address info. Add fields to DB transactions table.
- Add to explorerDataSourceAlt: AddressHistory and FillAddressTransactions to get a complete explorer.AddressInfo from a slice of *dbtypes.AddressRow.
- Update addressPage to use these functions when not in lite mode.
- Add BlockTime, Time, TxType, Size, Spent, Sent, and Fees to dbtypes.Tx and to the transactions table in dcrpg.
- Add cached height to ChainDB, updated when block stored.
- Add AddressHistory to ChainDB to query DB for []*dbtypes.AddressRows for a given address.
- Add FillAddressTransactions to fill out the fields of a passed *explorer.AddressTx by querying the transactions table for the needed info on each transaction.
- Add queries to retrieve full transaction by hash.
- Add queriy to retrieve all information for an address.
- Compute spent, sent, fees, for each dbtypes.Tx in processTransactions.
- Add ReduceAddressHistory to explorertypes to create an initialized but incomplete explorer.AddressInfo from a []*dbtypes.AddressRow.
Add txhelpers.VoteVersion to get a vote version from a pkScript []byte.
Set Confirmations and TotalUnconfirmed in FillAddressTransactions.
Document ReduceAddressHistory.
Add is_valid to blocks, block_height to transactions.
BlockHeight also added to dbtypes.Tx.
Fix Confirmations for address page, using BlockHeight instead of index.
Reorder transactions table columns.
Add spending tx info in RetrieveAddressTxns (fix), complete ReduceAddressHistory for spending.
Fix retrieve tx functions using dbtypes.UInt64Array.
* Add LIMIT and OFFSET suppport for address query.
Support or setting the limit and offset (count and start) from the URL is added using the "n" and "start" URL query parameters on the /address/{address} page.
Simplify (*wiredDB).GetExplorerAddress.
Unexport internal explorer param AddressRows -> maxAddressRows, and min/maxExplorerRows.
Add defaultAddressRows const.
Docs.
* a little extra space for the address on firefox
* Change --resumesync to --reindex in rebuilddb2.
* When adding block, set validity of previous block based on votes.
Add UpdateLastBlock query to set is_valid when needed.
Add Valid to explorer.BlockBasic, but it is currently always set to true.
* Smart sync mode with dcrdata.
Depending on how far the database is from the node's best block, the indexes may be dropped prior to sync, and the address table updates may be skipped in favor of a full rebuild after block/tx sync.
* Query and cache for total address row count.
The address page uses a LIMIT query behind the scenes, so for the address page to say how many are in total, a second query is used to count the rows. This is only necessary occasionally, so this adds the addressCounter type to store these counts.
New blocks invalidate (clear) the address counts and the queries must be executed again for fresh values.
explorer.AddressHistory now also returns the total number of funding/receiving transactions (total rows from the PSQL table).
Add KnownFundingTxns to explorer.AddressInfo and store the count from AddressHistory in it.
The dcrsqlite db (wiredDB) cannot get this count, so it is always 0 in the address page data structure when in lite mode.
Add NumFundingTxns and NumSpendingTxns to the explorer.AddressInfo type and set these values in both sqlite and pg dbs.
Remove AddressRow field.
* Handle starting from genesis on dcrdata
* Switch over to slow and safe sync after indexes are built. (bug fix)
* Update README with rebuilddb2 info and PostgreSQL notes.
* typo in address page template
* Remove space before single digit day number.
* Total address balance info with dcrpg.
New type explorer.AddressBalance.
Use AddressBalance in addressCounter map.
AddressHistory returns AddressBalance instead of just a tx count.
Add queries to get address row count and value sums for spent and unspent outputs.
Add Limit and Offset to explorer.AddressInfo since this type is used with limit/offset queries.
Show AddressBalance data on address page above table with limit.
Set ChainDB.bestBlock in constructor.
* bump to 1.0.0-pre
* Faster SelectAddressLimitNByAddress statement using subquery that allows use of address index.
* Fix nil pointer deref on address page for large start=X query.
* stakedb windows: do not wipe db if already opened.
* the other nil pointer deref
* Use different address query for dev subsidy since it is extremely common and a different query is faster.- Loading branch information
