Welcome to ncem’s documentation!

ncem

PyPI Python Version License Read the documentation at https://ncem.readthedocs.io/ Build Package Status pre-commit Black

ncem concept

Features

ncem is a model repository in a single python package for the manuscript Fischer, D. S., Schaar, A. C. and Theis, F. Learning cell communication from spatial graphs of cells. 2021. (preprint)

Installation

You can install ncem via pip from PyPI:

$ pip install ncem

Credits

This package was created with cookietemple using Cookiecutter based on Hypermodern_Python_Cookiecutter.

Installation

Stable release

To install ncem, run this command in your terminal:

$ pip install ncem

This is the preferred method to install ncem, as it will always install the most recent stable release.

If you don’t have pip installed, this Python installation guide can guide you through the process.

From sources

The sources for ncem can be downloaded from the Github repo. Please note that you require poetry to be installed.

You can either clone the public repository:

$ git clone git://github.com/theislab/ncem

Or download the tarball:

$ curl -OJL https://github.com/theislab/ncem/tarball/main

Once you have a copy of the source, you can install it with:

$ make install

API

Import ncem as:

import ncem

Estimator classes: estimators

Estimator classes from ncem for advanced use.

estimators.Estimator()

Estimator class for models.

estimators.EstimatorGraph()

EstimatorGraph class for spatial models.

estimators.EstimatorNoGraph()

EstimatorNoGraph class for baseline models.

estimators.EstimatorCVAE([use_type_cond, ...])

Estimator class for conditional variational autoencoder models.

estimators.EstimatorCVAEncem([cond_type, ...])

Estimator class for conditional variational autoencoder NCEM models.

estimators.EstimatorED([use_type_cond, ...])

Estimator class for encoder-decoder models.

estimators.EstimatorEDncem([cond_type, ...])

Estimator class for encoder-decoder NCEM models.

estimators.EstimatorInteractions([log_transform])

Estimator class for interactions models.

estimators.EstimatorLinear([log_transform])

Estimator class for linear models.

Model classes: models

Model classes from ncem for advanced use.

Classes that wrap tensorflow models.

models.ModelCVAE(input_shapes[, latent_dim, ...])

Model class for conditional variational autoencoder.

models.ModelCVAEncem(input_shapes[, ...])

Model class for NCEM conditional variational autoencoder with graph layer IND (MAX) or GCN.

models.ModelED(input_shapes[, latent_dim, ...])

Model class for non-spatial encoder-decoder.

models.ModelEDncem(input_shapes[, ...])

Model class for NCEM encoder-decoder with graph layer IND (MAX) or GCN.

models.ModelInteractions(input_shapes[, ...])

Model class for interaction model, baseline and spatial model.

models.ModelLinear(input_shapes[, l2_coef, ...])

Model class for linear model, baseline and spatial model.

Train: train

The interface for training ncem compatible models.

Trainer classes

Classes that wrap estimator classes to use in grid search training.

train.TrainModelCVAE()

train.TrainModelCVAEncem()

train.TrainModelED()

train.TrainModelEDncem()

train.TrainModelInteractions()

train.TrainModelLinear()

Grid search summaries

Classes to pool evaluation metrics across fits in a grid search.

train.GridSearchContainer(source_path, ...)

GridSearchContainer class.

Tutorials

We provide tutorials in separate repository.

  • A tutorial for fitting and evaluating a interactions model on the MERFISH - brain dataset (interactions).

If you would like to add more tutorials, feel free to contibute or open an issue.

Ecosystem

squidpy

squidpy provides an environment of tools that can be used to analysis spatial transcriptomnics in python.

scanpy

scanpy provides an environment of tools that can be used to analysis single-cell data in python.

Reference

Command-line interface.

Contributor Guide

Thank you for your interest in improving this project. This project is open-source under the BSD license and highly welcomes contributions in the form of bug reports, feature requests, and pull requests.

Here is a list of important resources for contributors:

How to add a dataloader

Overview of contributing dataloaders to ncem.

  1. Install ncem.

    Clone ncem into a local repository from development branch and install via pip.

cd target_directory
git clone https://github.com/theislab/ncem.git
git checkout development
# git pull  # use this to update your installation
cd ncem  # go into ncem directory
pip install -e .  # install
  1. Create a new dataloader in data.py

    Your dataloader should be a new class in data.py (ideally named by first author, e.g. DataLoaderZhang) and should contain the following functions _register_celldata, _register_img_celldata and _register_graph_features.

    _register_celldata creates an AnnData object called celldata of the complete dataset with feature names stored in celldata.var_names. Cell type annotations are stored in celldata.obs. celldata.uns[‘metadata’] should contain the naming conventions of files and columns in obs.

    _register_img_celldata then automatically splits the celldata into a dictionary of AnnData object with one AnnData object per image in the dataset.

    _register_graph_features can be added in case of additional graph features, e.g. disease status of images.

    Additionally, each dataloader should have a class attribute cell_type_merge_dict which provides a logic on how to merge cell types in any subsequent analysis. This can be helpful when considering datasets with fine cell type annotations and a coarser annotation is wanted.

  2. Make loader public (Optional).

    You can contribute the data loader to public ncem as code through a pull request. Note that you can also just keep the data loader in your local installation if you do not want to make it public.

# make sure you are in the top-level ncem directory from step 1
git add *
git commit  # enter your commit description
# Next make sure you are up to date with dev
git checkout development
git pull
git checkout YOUR_BRANCH_NAME
git merge development
git push  # this starts the pull request.

In any case, feel free to open an GitHub issue on on the Issue Tracker.

How to report a bug

Report bugs on the Issue Tracker.

How to request a feature

Request features on the Issue Tracker.

How to set up your development environment

You need Python 3.7+ and the following tools:

You can install them with:

$ pip install poetry nox nox-poetry

Install the package with development requirements:

$ make install

You can now run an interactive Python session, or the command-line interface:

$ poetry run python
$ poetry run ncem

How to test the project

Run the full test suite:

$ nox

List the available Nox sessions:

$ nox --list-sessions

You can also run a specific Nox session. For example, invoke the unit test suite like this:

$ nox --session=tests

Unit tests are located in the tests directory, and are written using the pytest testing framework.

How to submit changes

Open a pull request to submit changes to this project against the development branch.

Your pull request needs to meet the following guidelines for acceptance:

  • The Nox test suite must pass without errors and warnings.

  • Include unit tests. This project maintains a high code coverage.

  • If your changes add functionality, update the documentation accordingly.

To run linting and code formatting checks before committing your change, you can install pre-commit as a Git hook by running the following command:

$ nox --session=pre-commit -- install

It is recommended to open an issue before starting work on anything. This will allow a chance to talk it over with the owners and validate your approach.

Credits

Development Lead

Contributors

None yet. Why not be the first?

Contributor Covenant Code of Conduct

Our Pledge

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.

Our Standards

Examples of behavior that contributes to creating a positive environment include:

  • Using welcoming and inclusive language

  • Being respectful of differing viewpoints and experiences

  • Gracefully accepting constructive criticism

  • Focusing on what is best for the community

  • Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

  • The use of sexualized language or imagery and unwelcome sexual attention or advances

  • Trolling, insulting/derogatory comments, and personal or political attacks

  • Public or private harassment

  • Publishing others’ private information, such as a physical or electronic address, without explicit permission

  • Other conduct which could reasonably be considered inappropriate in a professional setting

Our Responsibilities

Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.

Scope

This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.

Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.

Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project’s leadership.

Attribution

This Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

Indices and tables