Skip to content

Commit

Permalink
feat(cache): can now declaire pair as not duplicates and save to cache
Browse files Browse the repository at this point in the history
  • Loading branch information
@theophanemayaud
theophanemayaud committed Nov 6, 2023
1 parent afa40b5 commit a7d4749
Show file tree
Hide file tree
Showing 6 changed files with 153 additions and 28 deletions.
12 changes: 11 additions & 1 deletion QtProject/app/comparison.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,9 @@ bool Comparison::bothVideosMatch(const Video *left, const Video *right)
return theyMatch;
}

// TODO check if pair is flagged as not dupplicate in DB
// check if pair is flagged as not dupplicate in DB
if(Db(_prefs.cacheFilePathName).isPairToIgnore(left->_filePathName, right->_filePathName))
return false;

_phashSimilarity = 0;

Expand Down Expand Up @@ -1412,3 +1414,11 @@ void Comparison::on_settingNamesInAnotherCheckbox_stateChanged(int arg1)
ui->label_namesContainedInOneAnotherStatus_autoIdentFiles->setText(status);
ui->label_namesContainedInOneAnotherStatus_autoOnlySizeDiff->setText(status);
}

void Comparison::on_ignoreDuplicatePairButton_clicked()
{
Db cache(_prefs.cacheFilePathName); // opening connexion to database
cache.writePairToIgnore(_videos[_leftVideo]->_filePathName, _videos[_rightVideo]->_filePathName);
on_nextVideo_clicked();
}

6 changes: 4 additions & 2 deletions QtProject/app/comparison.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ class Comparison : public QDialog

QVector<Video *> _videos;
Prefs &_prefs;
int _leftVideo = 0;
int _rightVideo = 0;
int _leftVideo = 0; // index in the video list, of the currently displayed left video
int _rightVideo = 0; // index in the video list, of the currently displayed right video
int _videosDeleted = 0;
int64_t _spaceSaved = 0;
bool _seekForwards = true;
Expand Down Expand Up @@ -164,6 +164,8 @@ private slots:

void on_pushButton_onlyTimeDiffersAutoTrash_clicked() {autoDeleteLoopthrough(AutoDeleteConfig(AUTO_DELETE_ONLY_TIMES_DIFF)); }

void on_ignoreDuplicatePairButton_clicked();

signals:
void sendStatusMessage(const QString &message) const;
void switchComparisonMode(const int &mode) const;
Expand Down
84 changes: 70 additions & 14 deletions QtProject/app/comparison.ui
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1099,20 +1099,76 @@
</layout>
</item>
<item>
<widget class="QPushButton" name="swapFilenames">
<property name="maximumSize">
<size>
<width>150</width>
<height>16777215</height>
</size>
</property>
<property name="layoutDirection">
<enum>Qt::LeftToRight</enum>
</property>
<property name="text">
<string>Swap filenames</string>
</property>
</widget>
<layout class="QHBoxLayout" name="swap_ignore_Layout" stretch="0,0,0,0,0">
<item>
<widget class="QPushButton" name="swapFilenames">
<property name="maximumSize">
<size>
<width>150</width>
<height>16777215</height>
</size>
</property>
<property name="layoutDirection">
<enum>Qt::LeftToRight</enum>
</property>
<property name="text">
<string>Swap filenames</string>
</property>
</widget>
</item>
<item>
<spacer name="horizontalSpacer_11">
<property name="orientation">
<enum>Qt::Horizontal</enum>
</property>
<property name="sizeHint" stdset="0">
<size>
<width>40</width>
<height>20</height>
</size>
</property>
</spacer>
</item>
<item>
<widget class="QPushButton" name="ignoreDuplicatePairButton">
<property name="toolTip">
<string>Mark this pair as not duplicates and save to the cache, so that it will not ask again until cache is cleared.</string>
</property>
<property name="text">
<string>Ignore</string>
</property>
</widget>
</item>
<item>
<spacer name="horizontalSpacer_9">
<property name="orientation">
<enum>Qt::Horizontal</enum>
</property>
<property name="sizeHint" stdset="0">
<size>
<width>40</width>
<height>20</height>
</size>
</property>
</spacer>
</item>
<item>
<spacer name="horizontalSpacer_7">
<property name="orientation">
<enum>Qt::Horizontal</enum>
</property>
<property name="sizeType">
<enum>QSizePolicy::Maximum</enum>
</property>
<property name="sizeHint" stdset="0">
<size>
<width>150</width>
<height>20</height>
</size>
</property>
</spacer>
</item>
</layout>
</item>
</layout>
</item>
Expand Down
63 changes: 53 additions & 10 deletions QtProject/app/db.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ void Db::emptyAllDb(const Prefs prefs){

query.exec(QStringLiteral("DROP TABLE IF EXISTS metadata"));
query.exec(QStringLiteral("DROP TABLE IF EXISTS capture"));
query.exec(QStringLiteral("DROP TABLE IF EXISTS ignored_pairs"));
query.exec(QStringLiteral("DROP TABLE IF EXISTS version"));

query.exec(QStringLiteral("VACUUM")); // restructure sqlite file to make it smaller on disk
Expand All @@ -102,16 +103,6 @@ void Db::emptyAllDb(const Prefs prefs){
QSqlDatabase().removeDatabase(connexionName); // clear the connexion backlog, basically... !
}

//QString Db::pathnameHashId(const QString &filename)
//{
// // Before : usesd file name and modified date, but could lead to two same identified files
// // if two files in seperate folders had the same name and modified date.
// // It was nice because even after moving files, they could still be identified
// // in the database.
// // Instead simply using full path and name, but we'd need to find a way to better uniquely
// // identify, that doesn't depend on path, to have file moving proofness !
// return QCryptographicHash::hash(filename.toLatin1(), QCryptographicHash::Md5).toHex();
//}
// -------------------- END : public static functions -------------------
// ----------------------------------------------------------------------

Expand Down Expand Up @@ -157,6 +148,19 @@ void Db::createTables(QSqlDatabase db, const QString appVersion)
");"
));

query.exec(QStringLiteral(
"CREATE TABLE IF NOT EXISTS "
"ignored_pairs ("
"pair_id INTEGER PRIMARY KEY AUTOINCREMENT, "
"pathName1 TEXT NOT NULL, "
"pathName2 TEXT NOT NULL, "
"FOREIGN KEY (pathName1) REFERENCES metadata(id), "
"FOREIGN KEY (pathName2) REFERENCES metadata(id), "
"UNIQUE (pathName1, pathName2) "
");"
));


// Now create a version table and entry, that could help us in the future to check if the database contains old records
// and might need to be emptied... ! For now, not used.
query.exec(QStringLiteral(
Expand Down Expand Up @@ -395,3 +399,42 @@ QStringList Db::getCachedVideoPathnamesInFolders(QStringList directoriesPaths) c

return videoPathNames;
}

void Db::writePairToIgnore(const QString filePathName1, const QString filePathName2) const {
QSqlQuery query(_db);
query.prepare("INSERT OR IGNORE INTO "
"ignored_pairs (pathName1, pathName2) "
"VALUES(:filePathName1, :filePathName2);");
query.bindValue(":filePathName1", filePathName1);
query.bindValue(":filePathName2", filePathName2);

if(!query.exec()){
qDebug() << "Error with writePairToIgnore query=" << query.lastQuery();
qDebug() << query.lastError().text();
}
}

bool Db::isPairToIgnore(const QString filePathName1, const QString filePathName2) const {
QSqlQuery query(_db);
query.prepare(
"SELECT * "
"FROM ignored_pairs "
"WHERE "
"(pathName1=:filePathName1 AND pathName2=:filePathName2) "
"OR "
"(pathName1=:filePathName2 AND pathName2=:filePathName1);"
);
query.bindValue(":filePathName1", filePathName1);
query.bindValue(":filePathName2", filePathName2);

if(!query.exec()){
qDebug() << "Error with isPairToIgnore query=" << query.lastQuery();
qDebug() << query.lastError().text();
return false;
}

if(query.next())
return true;
else
return false;
}
7 changes: 7 additions & 0 deletions QtProject/app/db.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,13 @@ class Db

// returns a list of unique cached video pathNames within specified folders
QStringList getCachedVideoPathnamesInFolders(QStringList directoriesPaths) const;

// when wanting to save a pair or videos to be ignored on next runs, this will save a record in cache
void writePairToIgnore(const QString filePathName1, const QString filePathName2) const;

// when wanting to check if a pair was previous saved to be ignored in cache
bool isPairToIgnore(const QString filePathName1, const QString filePathName2) const;

};

#endif // DB_H
9 changes: 8 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,8 @@ CutEnds compares the beginning and end of videos separately, trying to find matc

The app stores video information and thumnails in a cache, which makes it much faster the second time it is asked to scan a video. Videos are identified by their full path and name. Sometimes you might not want to use the cache if you have renamed some videos to the names of other videos, or if you are very low on disk space (cache can take a few hunded MB per thousands of scanned videos). You can also clear cache as explained below.

This setting does not apply to saved pairs declare as not duplicates.

- pHash / SSIM:

pHash is a fast and accurate algorithm for finding duplicate videos.
Expand All @@ -126,7 +128,7 @@ A threshold that is too low or too high will either display videos that don't ma

- Empty cache

(Under Tools) With this option you can empty the app cache when you want a fresh start for the next scans. It will also reduce the cache file size on disk, freeing up space.
(Under Tools) With this option you can empty the app cache when you want a fresh start for the next scans. It will also reduce the cache file size on disk, freeing up space. This will also delete all saved pairs declared as not duplicates.

- Select custom "move to" folder instead of trash, or restore default

Expand All @@ -142,6 +144,11 @@ A threshold that is too low or too high will either display videos that don't ma

These two buttons make it possible too move one of the videos to the other file's folder. This is useful if you want to re-organize your videos in a specific way.

- Ignore (button)

This button saves in cache an association of two videos, such that they will not be shown as a match anymore, regardless of other settings. Clearing cache resets those associations.


- Auto (tab)

Using this tab, you can choose some automatic removal modes and settings. Once you have configured one, and press on its button, the app will go through the list of all videos and move each matching one to trash/your selected folder, as per the mode's description.
Expand Down

0 comments on commit a7d4749

Please sign in to comment.