-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Import onyx state #49255
Merged
rlinoz
merged 60 commits into
Expensify:main
from
callstack-internal:feat/import-onyx-state
Oct 3, 2024
Merged
feat: Import onyx state #49255
Changes from 59 commits
Commits
Show all changes
60 commits
Select commit
Hold shift + click to select a range
f1d2200
fix: improve perf of isActionOfType by limiting calls to the includes
adhorodyski 6381821
feat(wip): draft the import-onyx-state perf debug functionality
adhorodyski f8e758b
chore: add comments describing the whole workflow in steps
adhorodyski 3b33ef2
fix: not iterable items from onyx in persisted requests, added web su…
kubabutkiewicz 69b1767
Merge branch 'main' of github.com:callstack-internal/Expensify-App in…
kubabutkiewicz caa2573
fix: make the android component more performant to not crash the app …
kubabutkiewicz 1fbe276
Merge branch 'main' of github.com:callstack-internal/Expensify-App in…
kubabutkiewicz b9e7373
Merge branch 'Expensify:main' into feat/import-onyx-state
adhorodyski 7d403da
Merge branch 'Expensify:main' into feat/import-onyx-state
adhorodyski 5c4d6ab
feat: add returning to original state
kubabutkiewicz 28937eb
Merge branch 'feat/import-onyx-state' of github.com:callstack-interna…
kubabutkiewicz 0fe5895
fix: clear onyx state before applying imported one
kubabutkiewicz 47926c0
fix: exporting onyx state on native
kubabutkiewicz 024b4a2
fix: refreshing page
kubabutkiewicz 02059bf
Merge branch 'main' into feat/import-onyx-state
TMisiukiewicz df69d08
align import button on web to be a menu item
TMisiukiewicz cc26c13
align import onyx state button on mobile as menu item
TMisiukiewicz f2c80c8
adjust transformNumericKeysToArray function usage
TMisiukiewicz 1588340
adjust web implementation
TMisiukiewicz fb3666b
adjust mobile import, cleanup mobile implementation
TMisiukiewicz 179bdf3
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz 71db305
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz 291402a
display only file upload on native
TMisiukiewicz 077432c
force offline when importing on mobile
TMisiukiewicz 65983c0
reduce diff across platforms
TMisiukiewicz 3397526
add onyx key to watch if state is imported
TMisiukiewicz b6f375a
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz 21205ae
make sure loader is hidden
TMisiukiewicz ce1ae7e
code cleanup
TMisiukiewicz dbc414a
add imported state button for resetting onyx state
TMisiukiewicz fc84004
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz d8102f9
fix crash when reportID is empty after clearing state
TMisiukiewicz 2dd1e91
fix lint
TMisiukiewicz e756a8b
code review updates
TMisiukiewicz 9a7a330
fix error when importing the file
TMisiukiewicz eba5ae0
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz 35fd5f1
migrate from withOnyx to useOnyx
TMisiukiewicz 072afdd
fix typecheck
TMisiukiewicz e55113b
Merge branch 'main' into feat/import-onyx-state
TMisiukiewicz 62be7da
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz 778934a
fix lint
TMisiukiewicz c3853c5
fix displaying messages and composer
TMisiukiewicz 29e7857
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz 8d91eb8
update App actions
TMisiukiewicz 10bbee3
fix onyx set error
TMisiukiewicz 80d11b3
don't restore sequential queue when going back from imported state
TMisiukiewicz 6f7ce2a
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz aeb02c3
properly get reportNameValuePairs
TMisiukiewicz 601085b
remove unnecessary rendering condition
TMisiukiewicz 8ed35d9
create base component and display modal when file is invalid
TMisiukiewicz ca55510
navigate to home on clear, do not import preferred theme
TMisiukiewicz 619414d
default reportID as string
TMisiukiewicz 7265c23
fix linter
TMisiukiewicz f41d5c5
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz b28e3f2
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz e152883
Merge remote-tracking branch 'upstream/main' into feat/import-onyx-state
TMisiukiewicz c6ff8e3
disable force offline when in import mode
TMisiukiewicz 6bdc95f
remove betas from omitted keys
TMisiukiewicz 3e305c3
fix data transformation in export
TMisiukiewicz 2790ba4
handle null value when masking data
TMisiukiewicz File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
import React from 'react'; | ||
import type {FileObject} from '@components/AttachmentModal'; | ||
import AttachmentPicker from '@components/AttachmentPicker'; | ||
import DecisionModal from '@components/DecisionModal'; | ||
import * as Expensicons from '@components/Icon/Expensicons'; | ||
import MenuItem from '@components/MenuItem'; | ||
import useLocalize from '@hooks/useLocalize'; | ||
import useResponsiveLayout from '@hooks/useResponsiveLayout'; | ||
import useThemeStyles from '@hooks/useThemeStyles'; | ||
|
||
function BaseImportOnyxState({ | ||
onFileRead, | ||
isErrorModalVisible, | ||
setIsErrorModalVisible, | ||
}: { | ||
onFileRead: (file: FileObject) => void; | ||
isErrorModalVisible: boolean; | ||
setIsErrorModalVisible: (value: boolean) => void; | ||
}) { | ||
const {translate} = useLocalize(); | ||
const styles = useThemeStyles(); | ||
const {isSmallScreenWidth} = useResponsiveLayout(); | ||
|
||
return ( | ||
<> | ||
<AttachmentPicker | ||
acceptedFileTypes={['text']} | ||
shouldHideCameraOption | ||
shouldHideGalleryOption | ||
> | ||
{({openPicker}) => { | ||
return ( | ||
<MenuItem | ||
icon={Expensicons.Upload} | ||
title={translate('initialSettingsPage.troubleshoot.importOnyxState')} | ||
wrapperStyle={[styles.sectionMenuItemTopDescription]} | ||
onPress={() => { | ||
openPicker({ | ||
onPicked: onFileRead, | ||
}); | ||
}} | ||
/> | ||
); | ||
}} | ||
</AttachmentPicker> | ||
<DecisionModal | ||
title={translate('initialSettingsPage.troubleshoot.invalidFile')} | ||
prompt={translate('initialSettingsPage.troubleshoot.invalidFileDescription')} | ||
isSmallScreenWidth={isSmallScreenWidth} | ||
onSecondOptionSubmit={() => setIsErrorModalVisible(false)} | ||
secondOptionText={translate('common.ok')} | ||
isVisible={isErrorModalVisible} | ||
onClose={() => setIsErrorModalVisible(false)} | ||
/> | ||
</> | ||
); | ||
} | ||
|
||
export default BaseImportOnyxState; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
import React, {useState} from 'react'; | ||
import RNFS from 'react-native-fs'; | ||
import Onyx from 'react-native-onyx'; | ||
import type {FileObject} from '@components/AttachmentModal'; | ||
import {KEYS_TO_PRESERVE, setIsUsingImportedState} from '@libs/actions/App'; | ||
import {setShouldForceOffline} from '@libs/actions/Network'; | ||
import Navigation from '@libs/Navigation/Navigation'; | ||
import type {OnyxValues} from '@src/ONYXKEYS'; | ||
import ROUTES from '@src/ROUTES'; | ||
import BaseImportOnyxState from './BaseImportOnyxState'; | ||
import type ImportOnyxStateProps from './types'; | ||
import {cleanAndTransformState} from './utils'; | ||
|
||
const CHUNK_SIZE = 100; | ||
|
||
function readFileInChunks(fileUri: string, chunkSize = 1024 * 1024) { | ||
const filePath = decodeURIComponent(fileUri.replace('file://', '')); | ||
|
||
return RNFS.exists(filePath) | ||
.then((exists) => { | ||
if (!exists) { | ||
throw new Error('File does not exist'); | ||
} | ||
return RNFS.stat(filePath); | ||
}) | ||
.then((fileStats) => { | ||
const fileSize = fileStats.size; | ||
let fileContent = ''; | ||
const promises = []; | ||
|
||
// Chunk the file into smaller parts to avoid memory issues | ||
for (let i = 0; i < fileSize; i += chunkSize) { | ||
promises.push(RNFS.read(filePath, chunkSize, i, 'utf8').then((chunk) => chunk)); | ||
} | ||
|
||
// After all chunks have been read, join them together | ||
return Promise.all(promises).then((chunks) => { | ||
fileContent = chunks.join(''); | ||
|
||
return fileContent; | ||
}); | ||
}); | ||
} | ||
|
||
function chunkArray<T>(array: T[], size: number): T[][] { | ||
const result = []; | ||
for (let i = 0; i < array.length; i += size) { | ||
result.push(array.slice(i, i + size)); | ||
} | ||
return result; | ||
} | ||
|
||
function applyStateInChunks(state: OnyxValues) { | ||
const entries = Object.entries(state); | ||
const chunks = chunkArray(entries, CHUNK_SIZE); | ||
|
||
let promise = Promise.resolve(); | ||
chunks.forEach((chunk) => { | ||
const partialOnyxState = Object.fromEntries(chunk) as Partial<OnyxValues>; | ||
promise = promise.then(() => Onyx.multiSet(partialOnyxState)); | ||
}); | ||
|
||
return promise; | ||
} | ||
|
||
export default function ImportOnyxState({setIsLoading, isLoading}: ImportOnyxStateProps) { | ||
const [isErrorModalVisible, setIsErrorModalVisible] = useState(false); | ||
|
||
const handleFileRead = (file: FileObject) => { | ||
if (!file.uri) { | ||
return; | ||
} | ||
|
||
setIsLoading(true); | ||
readFileInChunks(file.uri) | ||
.then((fileContent) => { | ||
const transformedState = cleanAndTransformState<OnyxValues>(fileContent); | ||
setShouldForceOffline(true); | ||
Onyx.clear(KEYS_TO_PRESERVE).then(() => { | ||
applyStateInChunks(transformedState).then(() => { | ||
setIsUsingImportedState(true); | ||
Navigation.navigate(ROUTES.HOME); | ||
}); | ||
}); | ||
}) | ||
.catch(() => { | ||
setIsErrorModalVisible(true); | ||
}) | ||
.finally(() => { | ||
setIsLoading(false); | ||
}); | ||
|
||
if (isLoading) { | ||
setIsLoading(false); | ||
} | ||
}; | ||
|
||
return ( | ||
<BaseImportOnyxState | ||
onFileRead={handleFileRead} | ||
isErrorModalVisible={isErrorModalVisible} | ||
setIsErrorModalVisible={setIsErrorModalVisible} | ||
/> | ||
); | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
import React, {useState} from 'react'; | ||
import Onyx from 'react-native-onyx'; | ||
import type {FileObject} from '@components/AttachmentModal'; | ||
import {KEYS_TO_PRESERVE, setIsUsingImportedState} from '@libs/actions/App'; | ||
import {setShouldForceOffline} from '@libs/actions/Network'; | ||
import Navigation from '@libs/Navigation/Navigation'; | ||
import type {OnyxValues} from '@src/ONYXKEYS'; | ||
import ROUTES from '@src/ROUTES'; | ||
import BaseImportOnyxState from './BaseImportOnyxState'; | ||
import type ImportOnyxStateProps from './types'; | ||
import {cleanAndTransformState} from './utils'; | ||
|
||
export default function ImportOnyxState({setIsLoading, isLoading}: ImportOnyxStateProps) { | ||
const [isErrorModalVisible, setIsErrorModalVisible] = useState(false); | ||
|
||
const handleFileRead = (file: FileObject) => { | ||
if (!file.uri) { | ||
return; | ||
} | ||
|
||
setIsLoading(true); | ||
const blob = new Blob([file as BlobPart]); | ||
const response = new Response(blob); | ||
|
||
response | ||
.text() | ||
.then((text) => { | ||
const fileContent = text; | ||
const transformedState = cleanAndTransformState<OnyxValues>(fileContent); | ||
setShouldForceOffline(true); | ||
Onyx.clear(KEYS_TO_PRESERVE).then(() => { | ||
Onyx.multiSet(transformedState) | ||
.then(() => { | ||
setIsUsingImportedState(true); | ||
Navigation.navigate(ROUTES.HOME); | ||
}) | ||
.finally(() => { | ||
setIsLoading(false); | ||
}); | ||
}); | ||
}) | ||
.catch(() => { | ||
setIsErrorModalVisible(true); | ||
setIsLoading(false); | ||
}); | ||
|
||
if (isLoading) { | ||
setIsLoading(false); | ||
} | ||
}; | ||
|
||
return ( | ||
<BaseImportOnyxState | ||
onFileRead={handleFileRead} | ||
isErrorModalVisible={isErrorModalVisible} | ||
setIsErrorModalVisible={setIsErrorModalVisible} | ||
/> | ||
); | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
type ImportOnyxStateProps = { | ||
isLoading: boolean; | ||
setIsLoading: (isLoading: boolean) => void; | ||
}; | ||
|
||
export default ImportOnyxStateProps; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
import cloneDeep from 'lodash/cloneDeep'; | ||
import type {UnknownRecord} from 'type-fest'; | ||
import ONYXKEYS from '@src/ONYXKEYS'; | ||
|
||
// List of Onyx keys from the .txt file we want to keep for the local override | ||
const keysToOmit = [ONYXKEYS.ACTIVE_CLIENTS, ONYXKEYS.FREQUENTLY_USED_EMOJIS, ONYXKEYS.NETWORK, ONYXKEYS.CREDENTIALS, ONYXKEYS.SESSION, ONYXKEYS.PREFERRED_THEME]; | ||
|
||
function isRecord(value: unknown): value is Record<string, unknown> { | ||
return typeof value === 'object' && !Array.isArray(value) && value !== null; | ||
} | ||
|
||
function transformNumericKeysToArray(data: UnknownRecord): UnknownRecord | unknown[] { | ||
const dataCopy = cloneDeep(data); | ||
if (!isRecord(dataCopy)) { | ||
return Array.isArray(dataCopy) ? (dataCopy as UnknownRecord[]).map(transformNumericKeysToArray) : (dataCopy as UnknownRecord); | ||
} | ||
|
||
const keys = Object.keys(dataCopy); | ||
|
||
if (keys.length === 0) { | ||
return dataCopy; | ||
} | ||
const allKeysAreNumeric = keys.every((key) => !Number.isNaN(Number(key))); | ||
const keysAreSequential = keys.every((key, index) => parseInt(key, 10) === index); | ||
if (allKeysAreNumeric && keysAreSequential) { | ||
return keys.map((key) => transformNumericKeysToArray(dataCopy[key] as UnknownRecord)); | ||
} | ||
|
||
for (const key in dataCopy) { | ||
if (key in dataCopy) { | ||
dataCopy[key] = transformNumericKeysToArray(dataCopy[key] as UnknownRecord); | ||
} | ||
} | ||
|
||
return dataCopy; | ||
} | ||
|
||
function cleanAndTransformState<T>(state: string): T { | ||
const parsedState = JSON.parse(state) as UnknownRecord; | ||
|
||
Object.keys(parsedState).forEach((key) => { | ||
const shouldOmit = keysToOmit.some((onyxKey) => key.startsWith(onyxKey)); | ||
|
||
if (shouldOmit) { | ||
delete parsedState[key]; | ||
} | ||
}); | ||
|
||
const transformedState = transformNumericKeysToArray(parsedState) as T; | ||
return transformedState; | ||
} | ||
|
||
export {transformNumericKeysToArray, cleanAndTransformState}; |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct me if I am wrong but it looks like we are overriding the promise from previous chunk with a promise from next chunk. Is this desired behaviour?
What happens with promises generated from chunks before the very last one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it creates a new promise that waits for the previous promise to resolve before applying the next chunk, creating a chain of promises. This ensures that the
Onyx.multiSet
operation for each chunk will execute in sequence, waiting for the previous operation to completeThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh right, that's nice!
Thanks for explanation ❤️