2021-03-01 - KDB Project Call Meeting Minutes

[alvin-c-shih]

Date

Monday, March 01, 2021 - 9:30am EST / 2:30pm BST

// First Monday of every month

Attendees

Full name	Affiliation	GitHub Username
Peter Gyorok	Morgan Stanley	@gyorokpeter
Alvin Shih	Morgan Stanley	@alvin-c-shih
Gary Davies	AquaQ	@drgdavies
Timofey Beletskiy	JPMorgan
Dmitry Zakharov	Deutsche Bank
Aaron Davies	Morgan Stanley	@adavies42
James McLeod	FINOS	@mcleo-d

Meeting notices

• FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.

• All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.

• FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact [email protected] with any questions.

• FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Minutes

• (5 min) Convene, roll call, welcome new people.
  • Display FINOS Antitrust Policy summary slide
  • Review Meeting Notices (see above)
• (5 min) Approve previous meeting minutes.
  • Approved by acclamation.
• (20 min) Meet maintainer of another fork of kdb Studio and discuss potential collaboration.
  • Implemented features listed here: https://github.com/dzmipt/kdbStudio/blob/master/notes.md
    • See Appendix A for qStudio roadmap.
  • Maintainer works for an organization that's a member of FINOS.
    • Hasn't gone through the internal onboarding workflow yet.
  • Looks like good opportunities to collaborate, but we'll wait until onboarding completes.
• (5 min) Kdb Studio updates?
  • Added source code style check.
  • Change log and road map from qStudio seem like good ideas.
    • Created corresponding issues. Change Log kdb-studio#48 Roadmap kdb-studio#49
• (15 min) Microbenchmarks and other recommended utilities?
  • Interest in I/O benchmarking tools for:
    • On-disk encryption options. CPU vs hardware-accelerated.
      • https://code.kx.com/q/kb/dare/
    • Benchmarking of AWS storage options.
    • Need some ideas for synthetic TAQ.
  • Memory usage for multithreaded processes?
    • Try calling psutil using embedPy.
      • See Appendix B below.
• (5 min) AOB, Q&A, Adjourn.
  • Interest in kdb in containers.
    • Seems like we need to develop a bit more familiarity before concrete collaboration can begin.
  • Alternative might be q API for BigQuery which is easier to scale.
    • Query Refinitiv's copy of the data instead of maintaining your own.
    • https://www.aquaq.co.uk/q/aquaquarantine-2021/
  • Some tweaks to the module loader happening to make bootstrapping easier.

Appendix A : Roadmap items for qStudio

• Import of QPad server configuration
• Drag&Drop in Server tree for moving branches
• Workspaces
  • loading configuration from different place
• Tabs:
  • pin tab
  • rename tabs
  • reshuffle tabs
• Result:
  • Details into result: server, duration, query
  • drill down into complex result output
  • Options to turn off q types in output
  • Console like output
• Send result output to a different server
  • [push table to different server as a new object]
• Customization
  • Dialog of Font/Colors for syntax highlighting
  • Customization of decimal, date and time formatting
• Text area:
  • word wrapping
  • text highlighting withing matched braces

Appendix B : Callling psutil using embedPy.

Code:

// Assumes embedPy has been initialized.
// https://www.selenic.com/smem/
// https://pypi.org/project/psutil/ (Unique Set Size)
// http://www.pybloggers.com/2016/02/psutil-4-0-0-and-how-to-get-real-process-memory-and-environ-in-python/
//
pim_psutil:.p.import`psutil
psutil_Process:pim_psutil[`:Process]
PFULL_ATTRS:`rss`vms`shared`text`lib`data`dirty`uss`pss`swap;

memory_full_info:{[pid]
  // Extracting attributes individually not exactly efficient.
  // But I don't trust that the tuple will come back in the
  //  same order for all future versions.
  PFULL_ATTRS!psutil_Process[pid][`:memory_full_info][]{x[y]`}/:hsym PFULL_ATTRS}

memory_percent:{[pid;memtype]
  psutil_Process[pid][`:memory_percent][`memtype pykw memtype]`}

Example output:

q)memory_full_info[.z.i]

rss   | 155291648
vms   | 3377815552
shared| 35561472
text  | 794624
lib   | 0
data  | 2589196288
dirty | 0
uss   | 120041472
pss   | 124615680
swap  | 0

q)memory_percent[.z.i;`pss]
0.01537376

WebEx info

Webex

• https://finos.webex.com/finos/j.php?MTID=mfc9190e818fd85f9e3ece5166f23d920

Dial-in and Project Materials

• US: +1-415-655-0003 US Toll
• UK: +44-20319-88141 UK Toll
• Access code: 663 746 304
• GitHub Repos:
  • https://github.com/finos/kdb
  • https://github.com/finos/kdb-studio
• Wiki Page:
  • https://finosfoundation.atlassian.net/wiki/spaces/DT/pages/329383945/kdb+Project
• KDB Metrics on FINOS
  • https://metrics.finos.org/goto/033b15c80f0d08cab70f40f4358932f6

[adavies42]

hi

[mcleo-d]

@alvin-c-shih, @gyorokpeter, Timofey and Gary,

It would be great if you could say hello in the comments 👍

James.

[alvin-c-shih]

G'day!