07 - Binary Transport

# Binary data transfer¶

### Motivation¶

Often for visualizations in genomics, massive social networks, or sensor data visualizations, it helps to be able to plot millions rather than simply hundreds of thousands of points.

By default, pydeck sends data from Jupyter to the frontend by serializing data to JSON. However, for massive data sets, the costs to serialize and deserialize this JSON can prevent a visualization from rendering.

In order to get around this, pydeck supports binary data transfer, which significantly reduces data size. Binary transfer relies on NumPy and its typed arrays, which are converted to JavaScript typed arrays and passed to deck.gl using precalculated binary attributes.

### Usage¶

Binary transport will only work if the following requirements are met:

- Use Jupyter. Binary transfer only works within Jupyter environments via `pydeck.bindings.deck.Deck.show`. It relies on the socket-level communication built into the Jupyter environment.
- Set `use_binary_transport`. `use_binary_transport` must be set to `True` explictly on your `Layer`.
- Pass data as a DataFrame. The `Layer`'s `data` argument must be a `pandas.DataFrame` object.
- Don't pass un-used columns. Data that is not intend to be rendered should not be passed into the `Layer`.
- Prefer columns of lists where possible. Accessor names must be strings representing column names within the `pandas.DataFrame`. For example, `get_position='position'` is correct, not `get_position=['x', 'y']`. For example, this data format, where `x` & `y` represent a position and `r`, `g`, and `b` represent color values, like this–

x | y | r | g | b
--- | --- | --- | --- | ---
0 | 1 | 0 | 0 | 0
0 | 5 | 255 | 0 | 0
5 | 1 | 255 | 255 | 0

should be converted to a nested format like this–

position | color
--- | ---
[0, 1] | [0, 0, 0]
[0, 5] | [255, 0, 0]
[5, 1] | [255, 255, 0]

## Example¶

To demonstrate this, we'll plot a series of nodes generated at random from the networkx library:

In [1]:

    import pydeck as pdk
    import pandas as pd

    NODES_URL = "https://raw.githubusercontent.com/ajduberstein/geo_datasets/master/social_nodes.csv"

    def generate_graph_data(num_nodes, random_seed):
        """Generates a graph of 10k nodes with a 3D force layout

        This function is unused but serves as an example of how the data in
        this visualization was generated
        """
        import networkx as nx  # noqa

        g = nx.random_internet_as_graph(num_nodes, random_seed)
        node_positions = nx.fruchterman_reingold_layout(g, dim=3)

        force_layout_df = pd.DataFrame.from_records(node_positions).transpose()
        force_layout_df["group"] = [d[1]["type"] for d in g.nodes.data()]
        force_layout_df.columns = ["x", "y", "z", "group"]
        return force_layout_df

We will use deck.gl's OrbitView (example here) which can help us plot these points in 3D:

The data we'll render is a 3D position with some categorical data (`group`):

In [2]:

    nodes = pd.read_csv(NODES_URL)
    nodes.head()

Out[2]:

| x | y | z | group
--- | --- | --- | --- | ---
0 | 0.021245 | -0.103758 | -0.007543 | T
1 | -0.099934 | -0.057345 | 0.047415 | T
2 | 0.007010 | 0.107698 | 0.006490 | T
3 | 0.076356 | -0.007087 | 0.008351 | T
4 | 0.030020 | -0.126868 | 0.022745 | T

We need this flattened data in a nested format, so we transform it:

In [3]:

    nodes["position"] = nodes.apply(lambda row: [row["x"], row["y"], row["z"]], axis=1)

    # Remove all unused columns
    del nodes["x"]
    del nodes["y"]
    del nodes["z"]
    nodes.head()

Out[3]:

| group | position
--- | --- | ---
0 | T | [0.021245069801807404, -0.10375810414552687, -...
1 | T | [-0.09993428736925124, -0.0573454238474369, 0....
2 | T | [0.007010015659034252, 0.10769829154014587, 0....
3 | T | [0.07635559886693953, -0.0070866537280380735, ...
4 | T | [0.0300203301012516, -0.12686751782894135, 0.0...

We'll also color the points by group and then normalize the colors since they are provided as floats–see the deck.gl documentation.

In [4]:

    colors = pdk.data_utils.assign_random_colors(nodes["group"])
    # Divide by 255 to normalize the colors
    # Specify positions and colors as columns of lists
    nodes["color"] = nodes.apply(
        lambda row: [c / 255 for c in colors.get(row["group"])], axis=1
    )
    # Remove unused column
    del nodes["group"]
    nodes.head()

Out[4]:

| position | color
--- | --- | ---
0 | [0.021245069801807404, -0.10375810414552687, -... | [0.0784313725490196, 0.19607843137254902, 0.88...
1 | [-0.09993428736925124, -0.0573454238474369, 0.... | [0.0784313725490196, 0.19607843137254902, 0.88...
2 | [0.007010015659034252, 0.10769829154014587, 0.... | [0.0784313725490196, 0.19607843137254902, 0.88...
3 | [0.07635559886693953, -0.0070866537280380735, ... | [0.0784313725490196, 0.19607843137254902, 0.88...
4 | [0.0300203301012516, -0.12686751782894135, 0.0... | [0.0784313725490196, 0.19607843137254902, 0.88...

In [5]:

    view_state = pdk.ViewState(
        offset=[0, 0],
        latitude=None,
        longitude=None,
        bearing=None,
        pitch=None,
        zoom=10
    )
    views = [pdk.View(type="OrbitView", controller=True)]

    nodes_layer = pdk.Layer(
        "PointCloudLayer",
        nodes,
        id="binary-points",
        use_binary_transport=True,
        get_position="position",
        get_normal=[10, 100, 10],
        get_color="color",
        pickable=True,
        auto_highlight=True,
        highlight_color=[255, 255, 0],
        radius=50,
    )

    pdk.Deck(layers=[nodes_layer], initial_view_state=view_state, views=views, map_provider=None).show()

### Confirming that data is being transferred over web sockets¶

You can open up your brower's developer console to confirm the data is being sent via binary transfer. In the console you should see a message like:

    >> transport.js:68 Delivering transport message
    >> binary: {"binary-points": {…}}
    >> transport: n {name: "Jupyter Transport (JavaScript <=> Jupyter Kernel)", _messageQueue: Array(0), userData: {…}, jupyterModel: n, jupyterView: n}
    type: "json-with-binary"

You should also be able to see the binary payload in your developer console's Network tab.