D3 has the reputation of being super-complicated because of all the libraries that are based on it, "simplifying" it so that everyone can use it. In the past year I wanted to create pretty unique type of data visualisation, so I dived into D3 and discovered it a makes a lot more sense than I though. Of course, if you only want a regular bar chart, you'll do better with things like C3, nvd3 etc'. But if you want anything a bit special, D3 itself is very powerful and the documentation in pretty good - there's no reason to avoid using it directly.
Definitely looking forward to try the new release.
To add to that, if you are a complete newbie to any data visualization, do not start with d3. If you want to make pretty charts programatically, using R/ggplot2 or Python/Seaborn is good enough. Even Excel is fine if you tweak the defaults.
D3 is good if your visualization benefits from interactivity, either with dynamic data adjustment or rich tooltips. But static visualizations are important too. (I recently restructured my workflow so I can output static images and interactive charts with the same code, which makes it the best of both worlds.)
What is your static+interactive workflow now, if I can ask? Also, is it fairly easy to build a workflow that generates static visualizations via D3 (i.e. making savable SVGs)?
I make charts with R/ggplot2. Standard workflow is to construct chart and save as static file. (PNG/SVG etc.) But with the plot.ly bridge, I can convert to an interactive chart w/ rich tooltips with very very little tweaking. (See this notebook: https://github.com/minimaxir/interactive-facebook-reactions/...)
To convert D3 to SVG, I believe all you need is a helper function but that is not my area of expertise.
Those aren't updated clientside (have to ask the server for the new chart instead of just the data), which is a different architecture that does not scale.
Indeed, what I like about plotly is that you can start pretty much anywhere (R, Python, Jupyter, browser JavaScript, node.js, Julia, Matlab...) and get to an interactive, explorable chart (disclaimer, I'm working on WebGL plotly functions eg. https://twitter.com/plotlygraphs/status/747879741399568384)
Broken actually does a lot of stuff client side and you can simply update the data. It's much closer to d3 for interactivity,and really quite powerful. It does lack some features ATM, but it continues to grow space.
Shiny supports either model, depending on the visualization. And also I think you have it backwards, if you have to scale to large amounts of data then you want to push a chart, not data, into the client.
I don't think your solution works for this use case since what he wants is to save the "XML serialized" SVG element instead of a "JSON serialized" JavaScript object.
A simple
d3.select('svg').node().outerHTML
would do the trick.
(You can use the same trick as in console-save to generate a download URL though.)
I have used Epoch[0] and Rickshaw[1] before for similar purposes, and have heard that Plotly works well too (especially if the process that is generating the data is Python-based).
For simple line charts, maybe Smoothie[2]?
Currently, I am trying to make a dashboard for visualizing many different data sources with WebSockets and Chartjs but it's still not quite ideal-- there's a bit of lag, and sending too much data via JSON really slows things down.
If anyone has solved that this problem I would appreciate it if they told me how they did it.
Plotly animations are coming along gradually, and Reactive Vega may be something to look at. Unrelated to these, I've been exploring real-time visualizations (and more in focus, real-time configurability) with D3 e.g. here: http://bl.ocks.org/monfera/35be5626a47110312c35 but Mike Bostock and many of others have technical notes on updating charts, like here: https://bost.ocks.org/mike/path/
I'm using Flot for https://podium.live, 10-20 charts with upwards of 1k points/chart updating at 10hz. It can work, but you need to do a few things like wrapping things in requestAnimationframe and not redrawing axes.
> Of course, if you only want a regular bar chart, you'll do better with things like C3, nvd3 etc'.
I know you probably know this, but for other people's sake - NVD3 requires the D3 library to work. It's therefore probably quite a good entry point into D3 - less complexity, but the same underlying code.
You know what's even easier than d3? Raw SVG elements. They're pretty powerful and the documentation is pretty good. There's no reason to avoid using it directly.
I've got to the same conclusion the last time I had to build some graph not quite standard. I first tried to grok d3, got lost in the sea of the possibility before finally resorted to putting svg rect at computed (x,y).
Fantastic work! D3 has a large API surface area, due to the functionality it covers. So even a relatively conservative major upgrade is a TON of work, and some of it might not have been pure fun only.
1) What's your opinion on the apparent case that the bulk of changes come from you, as society promotes teamwork so much? Is it a case of 'small team efficency'? (Btw. I know of others' contributions too, and reliance on Rollup, ColorBrewer, matplotlib/Viridis etc.)
2) What kept you going? It's probably at least a person-year of work just on your side that's not paid for by companies, not to mention some kind of startup bet. Have you alternatively toyed with the idea of starting a value-added layer, since so many for-profit organizations benefit from your work?
3) While it changes the API, and most everything is rewritten, it's IMO a conservative upgrade in that it doesn't steer away from familiar concepts and structure. It still does DOM binding, transitions, layout etc. fairly similarly. Have you considered much more disruptive departures? As an example, React's DOM diffing is more radically different from D3 selections than the change from D3 3.* to 4.* or the Grammar of Graphics as an API concept is more radically different from D3 than the 3.* -> 4.0 API changes. Or something like moving away from the 'functional objects' concept in favor of curried, fixed-argument pure functions, perhaps prefaced with the familiar chaining API would have been a large change (these are mostly examples rather than wishlist). What were YOUR ideas, had you wanted more disruptive changes, or 'D4'?
1. That’s a difficult question. There is an efficiency that comes from working with people you already know--maintaining a smaller set of relationships--rather than trying to manage a lot of small contributions from many. Jason Davies made huge contributions to D3 3.0 (in particular d3-geo), but alas I haven’t heard from him lately.
Most external contributions to D3 tend to be either bug fixes or new features. Bug fixes are often merged quickly, but new features require careful assessment. Each addition to the API has a complexity cost: it makes the library bigger; it has to be documented; it may instigate support questions and issues; it may have ripple effects on the development of other features. I’ve long taken Josh Bloch’s advice to heart and try to maximize the “power-to-weight” ratio of my APIs, and to try to keep them as small as possible while still expressive. Part of the goal of 4.0’s modularity is to make it easier for others to release their new features as standalone libraries that they can use with D3, rather than it needing to be part of “core”. There isn’t a “core” D3 anymore; there’s just a default bundle. (And if we ship a web-based tool for rolling your own bundle, the barrier will be even lower for plugins.)
2. Yes, I’ve been working on D3 full-time for the last year, since leaving The New York Times. Partly I am playing catchup. D3 has been more popular than I could have imagined, and I want to make sure that it’s as good as I can make it. I loved working for the NYT (and I miss my colleagues dearly), but it was hard to think deeply about the design of D3 when you’re on deadline. I am interested in finding a way to make D3 development self-sustaining financially, as I cannot do this for free indefinitely. Prior to launch today, I have been focused on shipping 4.0; now that’s it’s released, I am thinking more about what’s next.
3. Yes, it’s not a huge departure despite the many changes and the new implementation. I hope that’s a testament to the strength of the original design… or perhaps it’s just my confidence in it. (There is more different between Protovis and D3 than between D3 3.x and 4.0.) There is not only one right tool for visualization; the right tool depends on the application, such as whether you’re designing bespoke custom news graphics, exploring a dataset of the first time, or doing realtime monitoring of your network. D3 is intended to be the lowest layer of visualization tools: the visualization “kernel”, or “standard library”. You can build higher-level abstractions on top of D3 that is more tailored to your application. One of the areas I am considering exploring is higher-level abstraction, perhaps more along the lines of Grammar of Graphics.
I'm contributing to plotly.js right now, which is also declarative, and I see the pros and cons of levels of denotative semantics (mostly depends on goals and library user requirements) but it was just an example for what I mean by more radical changes. Good you mentioned it anyway, protovis -> D3 also represented a rather disruptive change. But perhaps I was late with these questions for this specific AMA.
Really excited about this release. It seems like nothing has been left untouched with major changes to everything from selections to layout generations and the modularisation of the entire lib (think lodash).
There are major namespace changes with basically everything being flattened ( d3.geom.voronoi -> d3.voronoi) and many of the typical patterns will have to be relearned (major changes to selections), so I don't see upgrading any existing projects and I may have to hold off on using 4.0 in new projects until I actually understand what has happened.
Aye, a lot has changed! I know change in itself is a pain; despite the quantity of changes I did try to avoid unnecessary change, and batch the changes into a rare major release. I believe such change is worthwhile, and that it’s better in the long run to keep improving the API than to be tied down to past mistakes.
The renames (like d3.voronoi instead of d3.geom.voronoi) are superficial changes, so although it might seem disruptive, it should be pretty easy to update your code and habits. At least, I haven’t found it difficult… The more design-y changes, like the new d3.stack and d3.treemap, require thought to migrate existing code. But the new designs are much better, so I hope you find the effort worthwhile.
I know Perl6 and Python3 have shown the dangers of changing things in disruptive ways. But I still appreciate your willingness to change things carefully to make them better. The length of time your software will be used in the future almost certainly outweighs the length of time it was used in the past :-)
I thought about the Python 2/3 debacle a lot this past year, and yes, I hope this isn’t a repeat. The Python migration was especially hard because of dependencies between libraries (you could only migrate your library or application to Python 3 after your dependencies migrated), and because Python is used for many large, long-term projects. Certainly D3, too, has its own plugin ecosystem and long-term projects, but it’s also commonly used as a standalone library in short-term work, such as in news graphics—where migration should be much easier. So hopefully that pain is lessened.
I think it is certainly worthwhile, and I've already used one of your new features in the d3-hierarchy package (d3.stratify) in a d3 v3 project. I think the general improvements are enough to warrant the breaking changes. I think you will see a brief period of a python 2/3 syndrome, but I think it will pass quickly.
Excellent work! I've been using bits and pieces of this since the start of the year and can attest to the fact that migration from 3 is relatively painless.
Just want to say how much I appreciate the refactoring into smaller libraries. d3 has enough traction that it's not something they had to do, but it's the right thing to do, and probably took a good deal of effort and thinking.
Thanks. It’s definitely more fun to work on the pieces of D3 individually, rather than feeling the weight of a massive monolith. Hopefully it will encourage others to develop more reusable libraries that depend on (the relevant parts of) D3, too.
Just wanted to say that your work is very much appreciated. I've been using D3 since it was Protovis and I love it. Excited to dig into the changes and try it out.
I don’t know about the “best” way, but there are quite a few ways people have explored so far; Google [d3 react] for starters… D3 4.0 introduces a new option which is to use the D3 minilibraries d3-scale, d3-shape, d3-hierarchy etc. without d3-selection, so that you manipulate the DOM exclusively through React. (Of course I’m biased and still recommend D3’s data-join and transitions for animating between views, but now there’s more flexibility.)
I agree, though I've found that the animation story with React for anything even mildly complex is still pretty terrible.
My day job is building big data visualization applications with lots of D3 and React, so I have a lot of opinions about this.
If you can get away with using CSS transitions or react-motion in your app, and you don't need D3's enter-update-exit selections, I feel React's declarative syntax is a much simpler mental model and is consistent with how the rest of my app works. But anything complex you'll need to drop out of React and build your DOM in D3, which has a very different notion of element lifecycles than React and doesn't allow you to reuse any of your other React components.
You don't really need anything special to integrate it with React, just a little wrapper component which calls your update method. This example isn't DRY but whatever, gives you the idea: https://gist.github.com/tj/38b04d87949c4dbc3bda4579a108b634
There are a number of libraries that bridge React and D3, but if you want to do it from scratch you have two primary options:
1. use D3's low-level components to calculate layouts then render directly as React components, typically SVG elements, e.x. https://jsfiddle.net/g3o6t9xh/1/
2. use React's component lifecycle methods to grab the component's DOM element and render full D3 visualizations, e.x. https://jsfiddle.net/6xw2wvbc/2/
I've been using option 1. (without React) to leverage d3's functionality on the server side, outputting templated SVGs after using a layout function to create transformations scalings etc. on the data. I've taken this approach on a few project over the last year and am confident it's a pretty good way forward, mirroring as it does the approach d3 takes for pie charts, treemaps and so on.
Code samples in the book still use D3 v3, but my most recent experiments have been in v4. There's a lot of content in there on animation and transitions, which seems like the hard part.
Currently figuring out how to do mass 10k+ element animations. Adding it to the book when I figure it out.
You'll probably want to use D3 to build a React component. The component provides the API to the rest of your application, while D3 takes over dataviz heavy lifting inside the component.
The most interesting aspect, which applies to apps beyond React, is likely to be respecting the D3 selections API[1], if needed. If you're already familiar with the ins and outs of selections, then I wouldn't expect any particular problems.
D3 is a relatively low-level library for composing custom visualizations. There are some good high-level abstractions on top that integrate well with React and provide specific kinds of preconfigured charts. You might want to check out Uber's React Vis components: https://github.com/uber/react-vis
I've recently been experimenting with pure SVG and React, which seems like a really good fit. Just draw SVG in your render function as naively as possible, and let React figure out the DOM transitions. Super easy to debug and reason about.
Not being too familiar with D3, what would I get by adopting it? I would hope for higher level primitives, but the API seems imperative and stateful, which doesn't seem like a good fit with React. Would I end up duplicating and syncing my existing app model state into D3 state?
We built a workflow design ui with svg under react and for the most part it is an absolute pleasure. Also pretty neat that you can treat the svg as an image/thumbnail. If you need the svg to be interactive, overlaying html works well too.
You can generally use them with 'is', for example: <marker id={'bar'} is markerHeight={viewModel.height} refX={viewModel.xCenter} markerWidth={viewModel.width}>
You get speed. For animation, doing a dom diff each frame is really slow. D3 is way way faster. For data visualization, d3 has a lot of well built extensions.
I tend to go in the other direction, I use d3 rather then react for the rest of the page, so I don't have any issues with integration.
The nice thing is that you can mix and match D3 components as needed, e.g. require certain layouts or other pure functions but use React in lieu of DOM binding (if that's preferred for one reason or another) and not use the D3 selections.
Does anyone know if there is a good way to do server-side rendering (svg files) of D3 charts while being able to reuse the same code in a browser?
We do a lot of charts at work to visualize electricity consumption and are currently thinking about how to rewrite the js we've accumulated over the years. I've found some attempts at doing server-side rendering but nothing really compelling for now...
D3 is a well thought out library to do interactive visualizations in the browser and having an easy way to do svg exports (for things like automated emails) would be a big plus. Doing screenshots through selenium ain't always so pleasant ;)
Not sure if it fits your use case, but; for one project I needed to have charts that the user could modify in their browser, and once they were happy they could hit 'download PDF' to get a PDF version.
Achieved this by (admittedly slightly clunky way) of sucking the SVG out of the chart and sending it to the server via ajax. Server intercepts SVG and passes it to inkscape which does pretty good command line svg -> pdf... then, sends back completed PDF.
One gotcha is that you won't get any CSS this way, so all styling has to be done via SVG props.
That's exactly what we are doing but running the SVG code through Apache Batik Rasterizer to turn it into png. I like this approach because the user can customise the chart in the front end and then the same resulting svg code is rendered by the backend. The only issue is styling as you mentioned.
> One gotcha is that you won't get any CSS this way, so all styling has to be done via SVG props.
There actually is a way to bake stylesheets directly into the SVG in a way that at least Inkscape can read, but it's tricky. We're successfully doing this in a graphing library tuned for genetic data that's built on top of d3 called LocusZoom.
The first step is to get the content of the stylesheet into a string. We do that here with a CORS request in case the stylesheet is being loaded over a CDN, as javascript won't be able to access the content of CDN-loaded stylesheets from document.styleSheets:
Since the stylesheet doesn't change once the page is loaded this only needs to happen once. From there, in our case, we have a dynamic/interactive SVG plot so the download needs to bundle up the current state of the SVG on request. Here's the method that does that:
Other than what no doubt looks familiar for pulling the raw SVG markup into a string we also insert a tag right after the opening <svg> tag that looks like this:
Where this.css_string is our stylesheet content. We then base64 encode the string and stick in the href of an <a> tag styled to look like a download button as seen here:
The end result is a download SVG link/button that is 100% client-side with our chosen stylesheet embedded.
Now, I have noticed that some styles defined in the stylesheet are valid in the browser but fail in InkScape and other SVG viewers. A prime example would be setting a stroke or fill to "transparent". InkScape doesn't interpret this properly and you can end up with solid black, just about the exact opposite of transparent. Keeping this in mind I stick to defining explicit "fill-opacity" and "stroke-opacity" values in the stylesheet for SVG elements and everything plays nice.
D3 is great for updating / augmenting a server generated chart on the client side, with one caveat. Usually it's good practice to do the D3 DOM binding with a key (similar to the react key, but is a function; second argument to .data()). This allows the proper identification of a data point to be identified when it comes to enter, update or exit. However, with server-generated contents, you don't have the __data__ properties on the DOM elements initially. So, before there's any addition/deletion/update on the client side, it's best to establish the binding and populate __data__. An alternative is, if your use case permits, to not use the key function and work off of DOM element sequence. Either way, the best resource IMO is the GUP I, II, III series from Mike: https://bl.ocks.org/mbostock/3808234
A really quick and easy way to do server-side rendering is to use a third-party hosted service such as https://www.url2png.com/ - set up a webapp somewhere that uses D3 to render your graph, then pass the URL to that page to url2png (using carefully selected height/width options) to render a .png version of your dynamic graph.
$29/month gets you up to 5,000 renders, so it's a very inexpensive option (compared to building and running your own server-side rendering tech).
1. Are there any plans to have a TypeScript version of the lib?
2. Would it be possible to add jsdoc comments for autocompletion?
3. Are the polyfills for Map and Set still necessary now that we have that in es2015?
4. Why is the lib so heavy on argument overloads?
5. Are there any code style restrictions? I often saw loops and conditions without curly brackets and alike.
6. Are there any fields you'd need support in for maintaining or reworking the library?
I really like D3.js although I haven't found an application for me yet. I find it hard to get/update the date from a none standard REST endpoint and display them, especially if they are kind of multichanneled, like working times of multiple users unordered.
I will update my tutorials and examples (and probably write new tutorials) in the near future, and I expect others will, too. But as I now have thousands of examples, I didn’t want to hold the release on updating all of them. My newer examples use 4.0 already, and I’ll be going through the others in the coming weeks.
Does this mean you intend to update your existing examples in place? It might be nice if you forked them with v4 instead to create a defacto upgrade guide.
I know many tutorials reference your examples and it might create more confusion if they are updated in place.
I suppose it depends on how much you want to keep supporting v3, but I'd argue its worth keeping the old knowledge base intact for a while.
Yep, I’m going to update my examples in-place. Consider it my heavy-handed encouragement to upgrade to 4.0. (For one thing, there are lot of 3.x bugs fixed in 4.0, and I have limited resources to provide free support to old versions.) My examples are backed by git repos, so you’ll still be able to find the 3.x versions if you need them.
Is it possible to have V3 and V4 coexisting in a project?
Like, adding some of the awesome v4 new things to an already working v3 project without having to rework all the v3 code to v4 compliance?
I've tried doing this with v4 beta, with strange behaviours :)
Hmm. It might be possible, though it doesn’t sound like a great idea to have both D3 3.x and the D3 4.0 default bundle loaded simultaneously, since there will be a lot of overlap. A better strategy might be to load just the new modules you want to use. Like if you just want the new force layout, then load d3-force: https://github.com/d3/d3-force
But it might be best to just migrate to 4.0. I know the changelist is long, but it might not be as hard as you think to migrate. And you can always send your questions to the d3-js Google group or d3.js on Stack Overflow and I’ll try to help.
On a project I work on* we've found that this technique has worked well so far. We're slowly moving from v3 to v4 by splitting out groups of components into their own packages and upgrading them as we go. We've been heavily inspired by d3's approach e.g. with the use of rollup for bundling.
We're probably at about 60% of the library upgraded/split-out but we've yet to tackle the more tricky changes like d3-selection.
I think it is quite possible in some cases due to the modularisation. I have already done that with in my D3 v3 projects. For example I used the d3.stratify method that is part of the d3-hierarchy module.
It's harder to get interaction done with canvas compared to SVG, but for large graphs the performance gains are huge. Curious to dive into the other new features.
Thanks for this incredible piece of software mbostock!
For those interested making d3.js reusable for data analysis, check out https://vida.io. We create templates out of d3.js visualizations with dynamic properties. You can create dashboards from custom d3.js components. For more examples, see https://vida.io/explore.
* New pretty continuous color schemes (viridis, rainbow, ColorBrewer)
* New axes that are easier to use.
* Vastly improved circle-packing layout.
* More stable, extensible force layout.
* Cleaner implementation of immutable selections & transitions.
And a ton of other stuff. Taken overall, it feels like a big step forward in robustness and cleanliness. And I’m hopeful that it’ll be easier for folks to develop more D3 modules.
For simpler need such as rendering a SVG map from Topojson or GeoJSON data, is there an alternative than D3 that can utilize React.js' rendering of SVG? With no animation, no charts, just map with zoom in and out and drag around, is there a benefit of using D3?
Did they remove the emphasis n the .enter() pattern yet? This is the weirdest thing compared to how just about every other library works, and it can totally be avoided by just using the data() stuff directly.
Definitely looking forward to try the new release.