Monitoring application for Cryo-EM data of the Max Planck Institute of Biochemistry.

This application reads data from from multiple microscopic recordings, when saved in .star files, from the filesystem without the need for a database and scales up to 100k and more files. A modern client optimized for desktop and mobile view analyzes and displays the data and images in realtime.

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

• 28.01

  • add: do not show outliers in miniplots
  • add: standard deviation in config
  • add: validate() .star-File
  • fix miniplots: deactivate zoom in mobile
  • fix mainplot: state is preserved after update
  • fix mainplot: deactivate legend & zoom in mobile
  • for debugging: print key when selecting row in table
  • fix Images: Title on hover
  • deactivate: image play/pause button in fullscreen view
  • deactivate: Fullscreen mode

• 08.01.

  • add: AutoDelete (hard drive)
  • various bug fixes
  • test: Windows ✓
  • test: Edge ✓
  • test: IE ×
  • test: CentOS ✓
  • Choose "All" in Plotmain

• 22.12.

  • fix Main plot: points/h
  • fix fullscreen mode

• 14.12.

  • Logging files: rotating days
  • fix webgl not showing plotdata
  • notification, if webgl is not supported
  • ANGLE flag -> readme
  • fix: safari compatibility
  • test on new linux copy
  • plot main: count/h

• 25.11.20

  • replace moment.js (deprecated) with date-fns
  • loading circle: orange + more thick
  • node version in Readme
  • description in data.star -> config
  • checked compression level
  • minplots: display "x% plotted" for subsets
  • fix: missing plotdata after fullscreen mode
  • fix: image display bugs
  • add: pull to reload on mobile
  • fix: mobile and layout bugs
  • add: miniplot value filter
  • rewrite: FileWatcher
  • host,port,root -> .env
  • clean: exit handling
  • add: log debug
  • times.star now extendable
  • new colors

• 16.11.20

  • API: minimal number of fetching
  • filter: client side -> server side
  • use compression
  • PlotMain: speed up
  • PlotMini: speed up + responsive
  • use minified plotly package
  • Fullscreen only calculated if opened
  • Mini histograms only calculated if opened
  • reduce fetch size 1/3 (no images.star)
  • set environment to "production"
  • aggregate values plotted, if too many
  • (remove data.star info text)
  • loading circle while fetching
  • Scroll to top at end of page
  • rebuild image display
  • new Filewatcher: watchman

• 08.11.20

  • fix table image still shown after changing page
  • remove header scroll
  • image loading parallelized + image error shown
  • added Test class
  • better structure in config
  • Test: 24.000 datapoints: ~7-8s

To run this application, you will need to install Node.js (^14.0).

• e.g. Ubuntu:

curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt-get install -y nodejs

• e.g. CentOs

dnf module install nodejs:14

• e.g. Mac:

brew install node

• e.g. Windows

  → use pre-built Windows Installer

Get, build and run:

#get and configure
git clone https://github.com/duc-ng/cryo-em-monitor.git
cd cryo-em-monitor
vim .env            #set ROOT_DATA to your data folder

#build
npm install
npm run build

#create test files
node test.js        #enter a number and exit with ctrl+c

#start
node app.js         #exit with ctrl+c

#or start with process manager (forever)
npm start
npm stop

→ The application can be opened at: http://localhost:5000

This application reads data directly from the filesystem into memory given this fixed folder and file structure:

~/ROOT_DATA/microscope/YYYY-MM-DD/key/file

e.g.

./data/Titan3/2020-12-04/91436631163808500/data.star

––– data                                  #root data directory
    ├── Titan1                            #one folder for each microscope
    ├── Titan2
    └── Titan3
        ├── 2020-11-17                    #one directory for each day
        ├── 2020-11-30
        ├── 2020-12-03
        └── 2020-12-04
            ├── 41801567764988104         #one directory per recording
            └── 91436631163808500
                ├── data.star             #contains all data values
                ├── times.star            #contains all recording time values
                ├── images.star           #contains names and descriptions of image files
                ├── ctfdiag.png
                ├── driftplot.png
                ├── motionCorrAvg.png
                ├── pick.png
                ├── psMotionCorrAvg.png
                ├── psRawAvg.png
                └── rawAvg.png

1. Configure local environment variables in ./.env
2. Configure app variables in ./src/config.json
3. Rebuild app: npm run build

This configuration is expandable.

→ This configuration is expandable. First value has to be set.

E.g. content of times.star:

data_procTimes

loop_

_mmsImageKey_Value
_mmsdateAuqired_Value
_mmsdateImported_Value
_mmsdateProcessed_Value
_mmsdateExported_Value
_mmsdateError_Value

 22401610646102390	2021-01-08-00:03:11	2021-01-08-00:03:11	2021-01-08-00:03:11	2021-01-08-00:03:11	0

→ This configuration is expandable.

E.g. content of data.star:

data_measure

loop_
_mmsImageKey_Value
_mmsmean_Value
_mmsdrift_Value
_mmsiciness_Value
_mmsdefocus_Value
_mmsresolution_Value
_mmsccOfCtfFit_Value
_mmsastigmatism_Value
_mmnumberOfParticles_Value

22401610646102390	0.19179381962081998	0.5578357710359079	0.3937294311401236	0.7571168150011385	0.3104739088494066	0.7018363759899893	0.6121199579849954	10

→ This configuration is expandable.

E.g. content of images.star:

data_images

loop_

_mmsImageKey_Value
_mmsrawAvg_Name
_mmsrawAvg_Info
_mmspsRawAvg_Name
_mmspsRawAvg_Info
_mmsmotionCorrAvg_Name
_mmsmotionCorrAvg_Info
_mmspsMotionCorrAvg_Name
_mmspsMotionCorrAvg_Info
_mmsdriftplot_Name
_mmsdriftplot_Info
_mmsctfdiag_Name
_mmsctfdiag_Info
_mmspick_Name
_mmspick_Info

 22401610646102390	rawAvg.png	"Raw Average"	psRawAvg.png	"PowerSpectrum of Raw Average"	motionCorrAvg.png	"MotionCorr Average"	psMotionCorrAvg.png	"PowerSpectrum of Motincorr Average"	driftplot.png	"Driftplot"	ctfdiag.png	"CtfFit Diagnostic"	pick.png	"Particle Positions"

Start react development server at: http://localhost:3000 (easier development with auto refresh after save and no need of building app after changes)

npm run dev                  #server & client (+auto refresh)

#or
npm run devclient            #client (+auto refresh)
npm run devserver            #server (+auto refresh)

Analyze app size:

npm run build
npm run analyze

Create test .star files:

node test.js

Production:

• build app with npm run build
• set NODE_ENV to production in .env

├── app.js                  #server
│   ├── FileWatcher.js
│   │   ├── Memory.js
│   │   └── Reader.js
│   ├── Logger.js
│   └── AutoDelete.js
│
└── App.js                  #client
    ├── site
    │   ├── Header.js
    │   ├── Footer.js
    │   ├── API.js
    │   └── Sidebar.js
    │       ├── Navigation.js
    │       ├── Filter.js
    │       ├── Sidebar.js
    │       └── Updates.js
    │
    ├── global
    │   ├── Data.js
    │   └── Theme.js
    │
    ├── content
    │   ├── Status.js
    │   │
    │   ├── ImageContainer.js
    │   │
    │   ├── TableContainer.js
    │   │   └── Table.js
    │   │       ├── TableToolbar.js
    │   │       │   └── TableExport.js
    │   │       ├── TableHeader.js
    │   │       └── TableRowSingle.js
    │   │
    │   ├── PlotMain.js
    │   │
    │   └── PlotsMini.js
    │       └── PlotsFullscreen.js
    │
    ├── utils
    │   ├── ContentContainer.js
    │   ├── FormatDate.js
    │   ├── ImageDisplay.js
    │   ├── ImageFullscreen.js
    │   ├── PullAndRefresh.js
    │   ├── ScrollToTop.js
    │   ├── DetectWebGL.js
    │   └── SmallDivider.js
    │
    └── assets
        └── logo.jpeg

• Data won't show up in plots (client).

  Some components rely on the WebGL API, which is mostly supported in modern browsers. Simply update your browser to the latest version, which supports WebGL. On older operating systems you may need to set your browser's ANGLE flag to support native OpenGL instead of WebGL.

• Application gets stuck in "npm run build".

  Add more RAM to your server environment.

• Node.js - JavaScript runtime environemnt
• React.js - Frontend framework
• Express.js - Backend framework
• Material-UI - UI component library
• Plot.ly - Interactive plots

• Duc Nguyen (Max Planck Institute of Biochemistry)